Performance Assessment for Total Laparoscopic Hysterectomy in the Operating Room: Validity Evidence of a Procedure-specific Rating Scale

Study objective: The technical conduct of total laparoscopic hysterectomy (LH) is critical to surgical outcomes. This study explored the validity evidence of an objective scale specific to the assessment of technical skills (H-OSATS) for 7 tasks of an LH with salpingo-oophorectomy procedure performed in the operating room.

Design: Observational cohort study.

Setting: Two academic hospitals in Marseille and Montpellier, France.

Patients: Three groups of operators (novice, intermediate, and experienced surgeons) were video recorded during their live performances of LH on a simple case. For each group, a dozen unedited videos were obtained for the following tasks: division of the round ligament, division of the infundibulopelvic ligament, creation of the bladder flap, opening of the posterior peritoneum, division of the uterine vessels, colpotomy, and closure of the vault.

Interventions: Two qualified raters blindly assessed each video using the H-OSATS rating scale. Inter-rater reliability and test-retest reliability were calculated as measures of internal structure. In a separate round of evaluations, the raters provided a global competent/noncompetent decision for each performance. As a measure of consequential validity, a pass/fail score was set for each task using the contrasting group method.

Measurements and main results: Three tasks (creation of the bladder flap, colpotomy, and closure of the vault) displayed sound validity evidence: a meaningful total score difference among the 3 groups of experience as well as between the intermediate and experienced surgeons, reliability outcomes of >0.7, and a pass/fail score with a theoretical false-positive rate of <10%.

Conclusion: The validity evidence of the H-OSATS rating scale differed for separate evaluations of the 7 tasks. Three tasks (i.e., creation of the bladder flap, colpotomy, and closure of the vault) revealed sound validity evidence, including at the level of the attending surgeon, whereas other tasks were more consistent with low-stakes formative evaluation standards.