Audio-digital recordings to assess ratings reliability in clinical trials of schizophrenia

We examined ratings reliability in 5 clinical trials of subjects with schizophrenia experiencing an acute exacerbation of psychosis. Audio-digital recordings of site-based interviews of the Positive and Negative Syndrome Scale (PANSS) or Brief Psychiatric Rating Scale (BPRS) were used to obtain blinded, site-independent scores to evaluate paired scoring concordance. High intraclass correlations were noted between 1810 paired site-based and site-independent PANSS scores (r = 0.801) and 1837 paired BPRS scores (r = 0.897) with high limits of agreement such that 93.9% of paired scores were within the calculated 95% confidence intervals. In 2 studies where sufficient PANSS data was available at baseline and endpoint, blinded site-independent ratings yielded a predictive value of 84.2% for replicating site-based response/nonresponse treatment outcomes. There was a significant positive correlation between site-based scores and paired scoring deviations (PANSS: r = 0.246; p < 0.0001; BPRS: r = 0.176; p < 0.0001). The magnitude (symptom severity) of PANSS or BPRS scores affected the directionality of paired scoring deviations in each study. Site-based raters scored the most symptomatic subjects higher and less symptomatic subjects lower than the paired site-independent raters on either instrument. This analysis affirms the utility of paired audio-digital scoring of site-based interviews as a surveillance strategy for schizophrenia studies. We noted a high predictive value of blinded site-independent raters to replicate site-based treatment outcomes. The bi-directionality of paired scoring deviations observed for both the PANSS and BPRS is consistent with findings found for depression rating instruments.