Evaluating the reliability and reproducibility of the Ottawa Thoracic Morbidity and Mortality classification system

Background: Minimizing adverse events after surgery is widely recognized as an important indicator of quality; yet no consensus has been reached on how to standardize the reporting of adverse events after surgical procedures. Our objectives were to develop a standardized classification system to monitor both the presence and severity of thoracic morbidity and mortality, and to evaluate its reliability and reproducibility among a national cohort of thoracic surgeons.

Methods: To assess the Thoracic Morbidity and Mortality classification system (based on the Clavien-Dindo classification of adverse events), a 31-item questionnaire was sent to all members of the Canadian Association of Thoracic Surgeons in August 2009, consisting of a general description of the Thoracic Morbidity and Mortality severity grades, 20 case-based questions of postoperative adverse events to be classified, and questions regarding personal judgments. We derived descriptive and quantitative information using weighted Kappa statistics.

Results: Fifty-two (54.7%) thoracic surgeons completed the questionnaire; 41 (78.8%) of the respondents were affiliated with an academic teaching hospital. A total of 1,326 individual weighted Kappa statistics were calculated for all distinct pairs of raters, of which 1,152 (87%) were greater than 0.81, a range that is interpreted as "almost perfect agreement." A further 174 (13%) were in the range between 0.61 and 0.8, interpreted as "substantial agreement." All results were statistically significant (p < 0.0001). The classification system was regarded as straightforward (98% of the respondents), reproducible (94%), logical (92%), and useful (98%).

Conclusions: The modified classification system appears to offer objective, reliable, and reproducible reporting of thoracic morbidity and mortality, and thus may assist continuous quality improvement in thoracic surgery.