A Bayesian hierarchical model for multi-level repeated ordinal data: analysis of oral practice examinations in a large anaesthesiology training programme

Oral practice examinations (OPEs) are used in many anaesthesiology programmes to familiarize anaesthesiology residents with the format of the oral examination administered by the American Board of Anesthesiology. The OPE outcome (final grade) consists of 'Definite Not Pass', 'Probable Not Pass', 'Probable Pass' and 'Definite Pass'. In our study to assess the validity of the OPE, residents took an average of two (ranging from one to six) OPEs, each of which was evaluated by two board certified anaesthesiologists randomly selected from a pool of 12. A key question of interest was to identify factors, for example, the length of training, didactic experience and other characteristics, that most influence OPE outcome. In addition, we were interested in assessing the reliability of the final grade, that is, the covariance parameters are of interest as well. However, estimating variance components in multi-level data with an unequal number of repeated ordinal outcomes presents several statistical challenges, such as how to estimate high dimensional random effects parameters, especially for ordinal outcomes. We propose a Bayesian hierarchical proportional odds model for data with such complexity. The flexibility of such a model allows us to make inference on the association of OPE outcomes with other factors and to estimate the variance components as well.