Inter-Rater Reliability for the Amputation Endpoint in the National Vascular Quality Initiative

Background: National quality reporting efforts after revascularization for peripheral artery disease (PAD) are ongoing. Validation of endpoints are necessary in national quality registries.

Objectives: This study sought to examine the interrater reliability for the endpoint of major amputation at 1 year in the Vascular Quality Initiative (VQI) registry and the Medicare-linked Vascular Quality Initiative registry (VQI-VISION) against electronic health record (EHR) review.

Methods: Surgical or endovascular revascularization procedures between January 1, 2010, and December 31, 2017, in the VQI registry and VQI-VISION for 2 academic health systems were queried. Major amputation data were abstracted by trained data collectors for the VQI and derived from Current Procedural Terminology codes for VQI-VISION. Cases underwent protocolized adjudication for the endpoint of major amputation by EHR review. Paired tests were used to evaluate the sensitivity and specificity. Spearman's ρ and Cohen's κ were used to evaluate interrater reliability.

Results: Amputation endpoints for 1,936 revascularizations were examined. Compared with major amputation data in EHR review, the sensitivity for the VQI registry was 35.9% and the specificity was 99.4% (ρ = 0.53; κ = 0.48). For VQI-VISION, sensitivity was 67.7% and specificity was 98.9% (ρ = 0.75; κ = 0.74). For any amputation in VQI data, sensitivity was 35.3% and specificity was 99.3% (ρ = 0.53; κ = 0.46), and for VQI-VISION, they were 71.6% and 97.7%, respectively (ρ = 0.75; κ = 0.74).

Conclusions: Almost two-thirds of the amputations in the VQI registry and one-third of amputations in VQI-VISION were missing at 1 year compared against adjudicated EHR review. In preparing for national reporting systems for major amputation tracking, data collection system reform is needed.