Psychometric assessment of the consideRATE questions, a new measure of serious illness experience, with an online simulation study

Objective: To assess the psychometric properties of the consideRATE questions, a measure of serious illness experience.

Methods: We recruited people at least 50 years old via paid panels online, with US-Census-based quotas. We randomized participants to a patient experience story at two time points. Participants completed a series of measures, including the consideRATE questions. We assessed convergent (Pearson's correlation), discriminative (one-way ANOVA with Tukey's test for multiple comparisons) and divergent (Pearson's correlation) validity. We also assessed intra-rater reliability (intra-class correlation) and responsiveness to change (t-tests).

Results: We included 809 individuals in our analysis. We established convergent validity (r = 0.77; p < 0.001); discriminative validity (bad/neutral stories [mean diff=0.4; p < 0.001]; neutral/ good stories [mean diff=1.3; p < 0.001]) and moderate divergent validity (r = 0.57; p < 0.001). We established sensitivity to change in all stories (bad/good [mean diff=1.52; p < 0.001]; good/bad [mean diff= -1.68; p < 0.001]; neutral/bad [mean diff= -0.57; p < 0.001]; good/neutral [mean diff= -1.11; p < 0.001]; neutral/good [mean diff= 1.1; p < 0.001]) but one (bad/neutral [mean diff= 0.4; p < 0.07]). Intra-rater reliability was demonstrated between time points (r = 0.77; p < 0.001).

Conclusions: the consideRATE questions were reliable and valid in a simulated online test.

Practice implications: the consideRATE questions may be a practical way to measure serious illness experience and the effectiveness of interventions to improve it.