A Validation Study of the Mindfulness-Based Interventions Teaching Assessment Criteria for Assessing Mindfulness-Based Intervention Teacher Skill: Inter-Rater Reliability and Predictive Validity

Frederick M Hecht; Rebecca S Crane; Patricia Moran; Willem Kuyken; Wendy Hartogensis; Judson Brewer

doi:10.1177/27536130241275962

A Validation Study of the Mindfulness-Based Interventions Teaching Assessment Criteria for Assessing Mindfulness-Based Intervention Teacher Skill: Inter-Rater Reliability and Predictive Validity

Glob Adv Integr Med Health. 2024 Sep 24:13:27536130241275962. doi: 10.1177/27536130241275962. eCollection 2024 Jan-Dec.

Authors

Frederick M Hecht¹, Rebecca S Crane², Patricia Moran¹, Willem Kuyken³, Wendy Hartogensis¹, Judson Brewer⁴

Affiliations

¹ Osher Center for Integrative Medicine, University of California, San Francisco, CA, USA.
² Centre for Mindfulness Research and Practice, Bangor University, Bangor, UK.
³ Oxford Mindfulness Centre, Oxford University, Oxford, UK.
⁴ Mindfulness Center at Brown, Brown University, Providence, RI, USA.

Abstract

Background: Prior data suggests the Mindfulness-Based Interventions: (MBI) Teaching Assessment Criteria (MBI:TAC) has good inter-rater reliability, but many raters knew teacher experience level.

Objective: We sought to further evaluate the MBI-TAC's inter-rater reliability and obtain preliminary data on predictive validity.

Methods: We videorecorded 21 MBSR teachers from academic and community settings. We trained 19 experienced MBI teachers in using the MBI:TAC. MBSR teachers were rated by three assessors; teachers and their assessors did not know one another. To assess predictive validity, MBSR students in courses taught by 18 of the MBSR teachers were invited to complete PROMIS-29 measures before the MBSR course, at the end of the course (month 2), and month 4.

Results: Intraclass correlation coefficients (ICCs) representing a single rater ranged from 0.33 to 0.56 on the 6 MBI:TAC domains. Using an average of two raters, ICC estimates ranged from 0.48 to 0.71 and ICCs generalizing to an average of three raters ranged from 0.6 to 0.8. Among n = 152 participating MBSR students, we found improvements from baseline to 2 months and 4 months in PROMIS measures of Anxiety, Depression, Fatigue, Sleep, and Social Role function (range in improvement 2.3 to 6.3, P < 0.0001 for all comparisons except Social Role at 2 months, P = 0.007). Higher MBI:TAC ratings were associated with greater improvements in anxiety among MBSR students from baseline to 2 months, with a -0.31 lower participant anxiety score per 1 unit increase in MBI:TAC composite teaching rating (95% CI -0.58, -0.05, P = 0.019), but we did not find statistically significant relationships with improvements in other PROMIS-29 domains.

Conclusions: ICCs indicated good reliability using an average of three ratings, but inter-rater reliability was only fair using a single rater. We found initial validation that higher MBI:TAC ratings predicted greater improvements in anxiety symptoms in MBSR participants.

Keywords: intervention fidelity; intervention integrity; mindfulness; mindfulness based stress reduction; mindfulness-based intervention teacher assessment criteria.