A comparison of scalable routine clinical materials and observer ratings to assess CBT fidelity

Amber Calloway; Torrey A Creed; Niki B Gumport; Cassidy Gutner; Luana Marques; Samantha Hernandez; Jiyoung Song; Clara Johnson; Soo Jeong Youn; Sohayla Elhusseini; Regine M Deguzman-Lucero; Taylor Laskot; Heidi La Bash; Yesenia Aguilar Silvan; Caroline Cassotte; Alayna L Park; Kimberlye Dean; Anna D Bartuska; Booil Jo; Paul Barnett; Eric Kuhn; Robert DeRubeis; Dawne Vogt; Shannon Wiltsey Stirman

doi:10.1016/j.brat.2024.104655

A comparison of scalable routine clinical materials and observer ratings to assess CBT fidelity

Behav Res Ther. 2024 Nov 26:184:104655. doi: 10.1016/j.brat.2024.104655. Online ahead of print.

Authors

Amber Calloway¹, Torrey A Creed², Niki B Gumport³, Cassidy Gutner⁴, Luana Marques⁵, Samantha Hernandez⁶, Jiyoung Song⁷, Clara Johnson⁸, Soo Jeong Youn⁹, Sohayla Elhusseini¹⁰, Regine M Deguzman-Lucero¹¹, Taylor Laskot¹¹, Heidi La Bash¹⁰, Yesenia Aguilar Silvan¹², Caroline Cassotte¹³, Alayna L Park¹⁴, Kimberlye Dean⁵, Anna D Bartuska¹⁴, Booil Jo³, Paul Barnett¹⁵, Eric Kuhn¹⁰, Robert DeRubeis¹⁶, Dawne Vogt¹⁷, Shannon Wiltsey Stirman¹⁰

Affiliations

¹ Penn Collaborative for CBT and Implementation Science, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA. Electronic address: [email protected].
² Penn Collaborative for CBT and Implementation Science, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA.
³ Department of Psychiatry and Behavioral Sciences, Stanford University, 401 Quarry Road, Stanford, CA, 94305, USA.
⁴ Department of Psychiatry, Boston University School of Medicine, Boston, MA, 02118, USA.
⁵ Department of Psychiatry, Massachusetts General Hospital, One Bowdoin Square, Boston, MA, 02114, USA.
⁶ Rutgers, The State University of New Jersey, 152 Frelinghuysen Road, Piscataway, NJ, 08854, USA.
⁷ Department of Psychology, University of California, Berkeley, 2121 Berkeley Way, Berkeley, CA, 94704, USA.
⁸ Department of Psychology, University of Washington, Guthrie Hall 119A, Seattle, WA, 98195, USA.
⁹ Reliant Medical Group, OptumCare, Harvard Medical School, 5 Neponset St, Worcester, MA, 01606, USA.
¹⁰ Department of Psychiatry and Behavioral Sciences, Stanford University, 401 Quarry Road, Stanford, CA, 94305, USA; National Center for PTSD, VA Palo Alto Healthcare System, 795 Willow Road (NC-PTSD), Menlo Park, CA, 94025, USA.
¹¹ National Center for PTSD, VA Palo Alto Healthcare System, 795 Willow Road (NC-PTSD), Menlo Park, CA, 94025, USA.
¹² Department of Psychology, University of California, Los Angeles (UCLA), 5505 Franz Hall, Los Angeles, CA, 90095, USA.
¹³ National Center for PTSD, VA Palo Alto Healthcare System, 795 Willow Road (NC-PTSD), Menlo Park, CA, 94025, USA; VA Boston Healthcare System, 150 S Huntington Ave, Boston, MA, 02130, USA.
¹⁴ University of Oregon, 1227 University St, Eugene, OR, 97403, USA.
¹⁵ Palo Alto Veterans Institute for Research, 3801 Miranda Avenue Palo Alto, CA, 94304, USA.
¹⁶ Department of Psychology, University of Pennsylvania, Philadelphia, PA, USA.
¹⁷ Department of Psychiatry, Boston University School of Medicine, Boston, MA, 02118, USA; National Center for PTSD, VA Palo Alto Healthcare System, 795 Willow Road (NC-PTSD), Menlo Park, CA, 94025, USA; Department of Psychology, University of California, Los Angeles (UCLA), 5505 Franz Hall, Los Angeles, CA, 90095, USA.

PMID: 39612724
DOI: 10.1016/j.brat.2024.104655

Abstract

Decades of research have demonstrated the efficacy of cognitive behavioral therapies (CBTs) for a wide variety of psychiatric diagnoses, resulting in the inclusion of CBT as a first-line evidence-based practice (EBP) in treatment guidelines for mood and anxiety disorders. However, some research suggests that many providers do not implement EBPs as intended. Ongoing quality monitoring is needed to support EBP implementation and sustainability, but "gold standard" fidelity monitoring (e.g. observer ratings) is time-consuming, requires extensive training, and may feel intrusive to providers and clients. In the current study, we aimed to develop a scalable method of assessing CBT fidelity that leverages information generated in routine clinical care (e.g. session worksheets and clinician checklists). Ratings of adherence based on worksheets were not correlated with ratings of adherence based on observer ratings. However, ratings of competence based on worksheets were significantly correlated with observer ratings of competence. Ratings of adherence based on clinician checklist ratings were also significantly correlated with observer-rated adherence. Results did not indicate a strong relationship between adherence or competence measured by worksheet ratings or observer and symptom change. However, adherence as measured by clinician checklists were associated with subsequent depression symptom change. Findings have a strong potential to impact fidelity monitoring strategies for a variety of CBTs. Given the limited resources and time to do full audio review in routine care settings, findings suggest that using routine materials generated in session to assess therapist competence may be a feasible alternative to the "gold standard" audio review. The trial is registered at ClinicalTrials.gov, number NCT03479398.

Keywords: Anxiety; Cognitive behavior therapy; Depression; Evidence-based; Fidelity; Implementation; Practices.

Associated data

ClinicalTrials.gov/NCT03479398