Observer variation in drug-induced sleep endoscopy: experienced versus nonexperienced ear, nose, and throat surgeons

Study objective: To determine variations in interobserver and intraobserver agreement of drug-induced sleep endoscopy (DISE) in a cohort of experienced versus nonexperienced ear, nose, and throat (ENT) surgeons.

Design: Prospective, blinded agreement study.

Setting: Ninety-seven ENT surgeons (90 nonexperienced with DISE; seven experienced) observed six different DISE videos and were asked to score the upper airway (UA) level (palate, oropharynx, tongue base, hypopharynx, epiglottis), direction (anteroposterior, concentric, lateral), and degree of collapse (none; partial or complete collapse). Findings were collected and analyzed, determining interobserver and intraobserver agreement [overall agreement (OA), specific agreement (SA)] and kappa values per UA level.

Measurement and results: In the nonexperienced group, overall interobserver agreement on presence of tongue base collapse (OA = 0.63; kappa = 0.33) was followed by the agreement on epiglottis (OA = 0.57; kappa = 0.23) and oropharynx collapse (OA = 0.45; kappa = 0.09). Low overall interobserver agreement in this group was found for hypopharyngeal collapse (OA = 0.33; kappa = 0.08). A similar ranking was found for degree of collapse. For direction of collapse, high interobserver agreement was found for the palate (OA = 0.57; kappa = 0.16). Among the experienced observers, overall interobserver agreement was highest for presence of tongue base collapse (OA = 0.93; kappa = 0.71), followed by collapse of the palate (OA = 0.80; kappa = 0.51). In this group, lowest agreement was also found for hypopharyngeal collapse (OA = 0.47; kappa = 0.03). Interob-server agreement on direction of collapse was highest for epiglottis collapse (OA = 0.97; kappa = 0.97). Concerning the degree of collapse, highest agreement was found for degree of oropharyngeal collapse (OA = 0.82; kappa = 0.66). Among the experienced observers a statistically significant higher interobserver agreement was obtained for presence, direction, and degree of oropharyngeal collapse, as well as for presence of tongue base collapse and degree of epiglottis collapse. Among the nonexperienced observers, high intraobserver agreement was found in particular for tongue base and epiglottis collapse. Among the experienced observers, high agreement was found for all levels but to a lesser extent for hypopharyngeal collapse. Intraobserver agreement was statistically significantly higher in the experienced group, for all UA levels expect for the hypopharynx.

Conclusion: This study indicates that both interobserver and intraobserver agreement was higher in experienced versus nonexperienced ENT surgeons. Agreement ranged from poor to excellent in both groups. The current results suggest that experience in performing DISE is necessary to obtain reliable observations.