Reliability and initial validation of the ulcerative colitis endoscopic index of severity

Background & aims: We studied the reliability of the previously described Ulcerative Colitis Endoscopic Index of Severity (UCEIS) and validated it with an independent cohort of investigators.

Methods: We created a new library of 57 videos of flexible sigmoidoscopy and stratified them based on disease severity. Twenty-five investigators were each randomly assigned to assess 28 videos (which included 4 duplicates to assess intraobserver reliability). Investigators were blinded to clinical details except for 2 of 4 duplicated videos (to assess the impact of knowledge of symptoms on assessment). Three descriptors ("vascular pattern", "bleeding", and "erosions and ulcers") comprising the UCEIS were scored with a visual analogue scale (VAS) to assess overall severity. Intrainvestigator and interinvestigator agreement was characterized by κ statistical analysis; reliability ratios were used to compare VAS and UCEIS scores.

Results: There was a high level of correlation between UCEIS scores and overall assessment of severity (correlation coefficient, 0.93). Internal consistency (Cronbach α analysis) was 0.86. Intrainvestigator and interinvestigator reliability ratios for UCEIS scores were 0.96 and 0.88, respectively. Intrainvestigator agreement in determination of the UCEIS score was good (κ = 0.72), with individual descriptors ranging from a κ of 0.47 (for bleeding) to 0.87 (for vascular pattern). Interinvestigator agreement in determination of UCEIS scores was moderate (κ = 0.50), with descriptors ranging from a κ of 0.48 (for bleeding) to 0.54 (for vascular pattern). Intrainvestigator variability in determining UCEIS scores did not change appreciably when a video was presented with clinical details.

Conclusions: The UCEIS and its components show satisfactory intrainvestigator and interinvestigator reliability. Among investigators, the UCEIS accounted for a median of 86% of the variability in evaluation of overall severity on the VAS when assessing the endoscopic severity of UC and was unaffected by knowledge of clinical details.