Reliability of the ECHOWS Tool for Assessment of Patient Interviewing Skills

Jill S Boissonnault; Kerrie Evans; Neil Tuttle; Scott J Hetzel; William G Boissonnault

doi:10.2522/ptj.20150172

Reliability of the ECHOWS Tool for Assessment of Patient Interviewing Skills

Phys Ther. 2016 Apr;96(4):443-55. doi: 10.2522/ptj.20150172. Epub 2015 Aug 27.

Authors

Jill S Boissonnault¹, Kerrie Evans², Neil Tuttle³, Scott J Hetzel⁴, William G Boissonnault⁵

Affiliations

¹ J.S. Boissonnault, PT, PhD, WCS, Department of Physical Therapy and Health Care Sciences, The George Washington University School of Medicine and Health Sciences, 2000 Pennsylvania Ave NW, Suite 227, Washington, DC 20006 (USA). Dr Boissonnault was affiliated with the Department of Orthopedics and Rehabilitation, Physical Therapy Program, University of Wisconsin-Madison, Madison, Wisconsin, at the time of the study. [email protected].
² K. Evans, PT, PhD, School of Allied Health Sciences, Menzies Health Institute Queensland, Griffith University-Gold Coast Campus, Gold Coast, Australia.
³ N. Tuttle, PT, PhD, School of Allied Health Sciences, Menzies Health Institute Queensland, Griffith University-Gold Coast Campus.
⁴ S.J. Hetzel, MS, Department of Biostatistics and Medical Informatics, University of Wisconsin-Madison.
⁵ W.G. Boissonnault, PT, DPT, DHSc, FAAOMPT, FAPTA, Executive Vice President, Professional Affairs, American Physical Therapy Association, 1111 N Fairfax St, Alexandria, Virginia. At the time of the study, Dr Boissonnault was affiliated with the Department of Orthopedics and Rehabilitation, Physical Therapy Program, University of Wisconsin-Madison.

PMID: 26316533
DOI: 10.2522/ptj.20150172

Abstract

Background: History taking is an important component of patient/client management. Assessment of student history-taking competency can be achieved via a standardized tool. The ECHOWS tool has been shown to be valid with modest intrarater reliability in a previous study but did not demonstrate sufficient power to definitively prove its stability.

Objective: The purposes of this study were: (1) to assess the reliability of the ECHOWS tool for student assessment of patient interviewing skills and (2) to determine whether the tool discerns between novice and experienced skill levels.

Design: A reliability and construct validity assessment was conducted.

Methods: Three faculty members from the United States and Australia scored videotaped histories from standardized patients taken by students and experienced clinicians from each of these countries. The tapes were scored twice, 3 to 6 weeks apart. Reliability was assessed using interclass correlation coefficients (ICCs) and repeated measures. Analysis of variance models assessed the ability of the tool to discern between novice and experienced skill levels.

Results: The ECHOWS tool showed excellent intrarater reliability (ICC [3,1]=.74-.89) and good interrater reliability (ICC [2,1]=.55) as a whole. The summary of performance (S) section showed poor interrater reliability (ICC [2,1]=.27). There was no statistical difference in performance on the tool between novice and experienced clinicians.

Limitations: A possible ceiling effect may occur when standardized patients are not coached to provide complex and obtuse responses to interviewer questions. Variation in familiarity with the ECHOWS tool and in use of the online training may have influenced scoring of the S section.

Conclusion: The ECHOWS tool demonstrates excellent intrarater reliability and moderate interrater reliability. Sufficient training with the tool prior to student assessment is recommended. The S section must evolve in order to provide a more discerning measure of interviewing skills.

Publication types

Research Support, Non-U.S. Gov't
Validation Study

MeSH terms

Australia
Clinical Competence*
Educational Measurement
Humans
Interviews as Topic / standards*
Medical History Taking / standards*
Physical Therapy Specialty / education
Reproducibility of Results
Students, Health Occupations*
United States