Grading reflective essays: the construct validity and reliability of a newly developed Tool- GRE-9

Background: The main objective of this study is to assess the construct validity and reliability of the Grading Reflective Essays-9 (GRE-9).

Methods: This study took place in a major tertiary academic medical center in Beirut, Lebanon. 104 reflective essays written by years 1-3 residents in the department of Family Medicine at the American University of Beirut Medical Center (AUBMC) were graded by 2 trained raters who independently scored the essays using GRE-9. GRE-9 scores were then correlated with scores on communication skills OSCE stations and in-training examinations to investigate, respectively, convergent and divergent validity. One of the 2 raters scored the essays twice one month apart to assess the reliability of the GRE-9 using intra rater reliability and internal consistency.

Results: There was a weak, non-significant correlation between GRE-9 score and In training examination (ITE) score (r = - .213, p = .395). There was a moderate, non-significant correlation between GRE-9 scores and the Objective structured clinical examination (OSCE) communication station scores (r = - .412 p = .162). The correlation coefficient between trails 1 and 2 was significant (r = .832, p = .000). Intra class correlation coefficient (ICC) analysis demonstrated almost perfect intra-rater agreement (0.819; 95% CI: 0.741-0.875) of the test ratings over time.

Conclusions: GRE-9, is a short, concise, easy-to-use reliable grading tool for reflective essays that has demonstrated moderate to substantial intra-rater reliability and evidence of divergent validity. The study found non-significant correlations between reflective writing scores OSCE communication scores demonstrating a lack of relationship between reflective writing and this measure of performance.