In order to compare the quality of CPR performance after various training methods, training outcome assessment must provide meaningful data and do it in a way that is reliable. Few studies have provided details of their assessment procedures, and even fewer report on whether the measures to evaluate performance are reliable (yielding information consistently over multiple trials), or valid (measuring the outcome intended). Few studies have attempted to replicate assessment methods used by other authors. Conventional skill sheets have not been shown to assess compressions and ventilations reliably and validly. When using an instrumented manikin, skill checklists can be simplified by eliminating qualitative assessment of compressions and ventilations. Using a sample of 171 CPR trainees rated by trained evaluators, we provide details of agreement between two evaluators and use an established statistic (Cronbach's alpha) to assess the reliability of a 14-item simplified CPR checklist. The level of agreement between two raters was high (Pearson product-moment correlation = 0.87) as was the reliability estimate obtained by Cronbach's alpha (0.89). As criterion-related evidence of the validity of the CPR checklist to assess CPR performance, a correlation with a five-point subjective overall rating of CPR was estimated (Spearman correlation = 0.92). We urge standardized reporting of CPR training outcomes in order to achieve comparability across studies.