Are Improvements in Measured Performance Driven by Better Treatment or "Denominator Management"?

Background: Process measures of healthcare quality are usually formulated as the number of patients who receive evidence-based treatment (numerator) divided by the number of patients in the target population (denominator). When the systems being evaluated can influence which patients are included in the denominator, it is reasonable to wonder if improvements in measured quality are driven by expanding numerators or contracting denominators.

Objective: In 2003, the US Department of Veteran Affairs (VA) based executive compensation in part on performance on a substance use disorder (SUD) continuity-of-care quality measure. The first goal of this study was to evaluate if implementing the measure in this way resulted in expected improvements in measured performance. The second goal was to examine if the proportion of patients with SUD who qualified for the denominator contracted after the quality measure was implemented, and to describe the facility-level variation in and correlates of denominator contraction or expansion.

Design: Using 40 quarters of data straddling the implementation of the performance measure, an interrupted time series design was used to evaluate changes in two outcomes.

Participants: All veterans with an SUD diagnosis in all VA facilities from fiscal year 2000 to 2009.

Main measures: The two outcomes were 1) measured performance-patients retained/patients qualified and 2) denominator prevalence-patients qualified/patients with SUD program contact.

Key results: Measured performance improved over time (P < 0.001). Notably, the proportion of patients with SUD program contact who qualified for the denominator decreased more rapidly after the measure was implemented (p = 0.02). Facilities with higher pre-implementation denominator prevalence had steeper declines in denominator prevalence after implementation (p < 0.001).

Conclusions: These results should motivate the development of measures that are less vulnerable to denominator management, and also the exploration of "shadow measures" to monitor and reduce undesirable denominator management.