Assessing Physician and Patient Agreement on Whether Patient Outcomes Captured in Clinical Progress Notes Reflect Treatment Success: Cross-Sectional Study

Background: It remains unclear if there is agreement between physicians and patients on the definition of treatment success following orthopedic treatment. Clinical progress notes are generated during each health care encounter and include information on current disease symptoms, rehabilitation progress, and treatment outcomes.

Objective: This study aims to assess if physicians and patients agree on whether patient outcomes captured in clinical progress notes reflect a successful treatment outcome following orthopedic care.

Methods: We performed a cross-sectional analysis of a subset of clinical notes for patients presenting to a Level-1 Trauma Center and Regional Health System for follow-up for an acute proximal humerus fracture (PHF). This study was part of a larger study of 1000 patients with PHF receiving initial treatment between 2019 and 2021. From the full dataset of 1000 physician-labeled notes, a stratified random sample of 25 notes from each outcome label group was identified for this study. A group of 2 patients then reviewed the sample of 100 clinical notes and labeled each note as reflecting treatment success or failure. Cohen κ statistics were used to assess the degree of agreement between physicians and patients on clinical note content.

Results: The average age of the patients in the sample was 67 (SD 13) years and 82% of the notes came from female patients. Patients were primarily White (91%) and had Medicare insurance coverage (65%). The note sample came from fracture-related encounters ranging from the second to the tenth encounter after the index PHF visit. There were no significant differences in patient or visit characteristics across concordant and discordant notes labeled by physicians and patients. Among agreement levels ranging from poor to perfect agreement, physician and patient evaluators exhibited only a fair level of agreement in what they deemed as treatment success based on a Cohen κ of 0.32 (95% CI 0.10-0.55; P=.01). Furthermore, interpatient and interphysician agreement also demonstrated relatively low levels of agreement.

Conclusions: The findings suggest that physicians and patients demonstrated low levels of agreement when assessing whether a patient's clinical note reflected a successful outcome following treatment for a PHF. As low levels of agreement were also observed within physician and patient groups, it is clear the definition of success varied highly across both physicians and patients. Further research is needed to elucidate physician and patient perceptions of treatment success. As outcome measurement and demonstrating the value of orthopedic treatment remain important priorities, it is important to better define and reach a consensus on what treatment success means in orthopedic medicine.