The impact of different data handling strategies in exploratory and confirmatory factor analysis of diary measures: an evaluation using simulated and real-world asthma nighttime symptoms diary data

Background: Daily diaries are an important modality for patient-reported outcome assessment. They typically comprise multiple questions, so understanding their underlying structure is key to appropriate analysis and interpretation. Structural evaluation of such measures poses challenges due to the high volume of repeated measurements. Potential strategies include selecting a single day, averaging item-level observations over time, or using all data while accounting for its multilevel structure.

Method: The above strategies were evaluated in a simulated dataset via exploratory and confirmatory factor modelling by comparing their impact on various estimates (i.e., inter-item correlations, factor loadings, model fit). Each strategy was additionally explored using real-world data from an observational study (the Asthma Nighttime Symptoms Diary).

Results: Both single day and item average strategies resulted in biased factor loadings. The former displayed lower overall bias (single day: 0.064; item average: 0.121) and mean square error (single day: 0.007; item average: 0.016) but greater frequency of incorrect factor number identification compared with the latter (single day: 46.4%; item average: 0%). Increased estimated inter-item correlations were apparent in the item-average method. Non-trivial between- and within-person variance highlighted the utility of a multilevel approach. However, convergence issues and Heywood cases were more common under the multilevel approach (90.2% and 100.0%, respectively).

Conclusions: Our findings suggest that a multilevel approach can enhance our insight when evaluating the structural properties of daily diary data; however, implementation challenges still remain. Our work offers guidance on the impact of data handling decisions in diary assessment.