Validity of task-specific metrics for assessment in perineal proctectomy

Background: Perineal proctectomy is a complex procedure that requires advanced skills. Currently, there are no simulators for training in this procedure. As part of our objective of developing a virtual reality simulator, our goal was to develop and validate task-specific metrics for the assessment of performance for this procedure. We conducted a three-phase study to establish task-specific metrics, obtain expert consensus on the appropriateness of the developed metrics, and establish the discriminant validity of the developed metrics.

Methods: In phase I, we utilized hierarchical task analysis to formulate the metrics. In phase II, a survey involving expert colorectal surgeons determined the significance of the developed metrics. Phase III was aimed at establishing the discriminant validity for novices (PGY1-3) and experts (PGY4-5 and faculty). They performed a perineal proctectomy on a rectal prolapse model. Video recordings were independently assessed by two raters using global ratings and task-specific metrics for the procedure. Total scores for both metrics were computed and analyzed using the Kruskal-Wallis test. A Mann-Whitney U test with Benjamini-Hochberg correction was used to evaluate between-group differences. Spearman's rank correlation coefficient was computed to assess the correlation between global and task-specific scores.

Results: In phase II, a total of 23 colorectal surgeons were recruited and consensus was obtained on all the task-specific metrics. In phase III, participants (n = 22) included novices (n = 15) and experts (n = 7). There was a strong positive correlation between the global and task-specific scores (rs = 0.86; P < 0.001). Significant between-group differences were detected for both global (χ² = 15.38; P < 0.001; df = 2) and task-specific (χ² = 11.38; P = 0.003; df = 2) scores.

Conclusions: Using a biotissue rectal prolapse model, this study documented high IRR and significant discriminant validity evidence in support of video-based assessment using task-specific metrics.