Concordance between independent first-line assessors in the Western Australian Audit of Surgical Mortality

Background: The Western Australian Audit of Surgical Mortality independently reviews deaths under a surgeon. A surgical case form completed by the surgeon is reviewed by a first-line assessor, who in 15% of cases recommends a detailed case note review. The aim of this study was to assess concordance between the original first-line assessment (FLA1) and an additional first-line assessment (FLA2) in relation to the necessity for a second-line assessment (SLA) and recognition of clinical management issues.

Methods: Experienced assessors (n = 39) representing different specialties were invited to be FLA2 assessors for 170 random cases. They completed identical FLA forms. FLA2 assessors were consultant surgeons with experience of the peer review process. Inter-rater reliability was determined using Gwet's Agreement Coefficient.

Results: A total of 170 FLAs, 158 (92.9%) were returned. Concordance between assessors for recommending an SLA was 70.5% (110 of 156), with an SLA recommended for 16 (10.3%) FLAs but not for 94 (60.3%) FLAs. When there was non-concordance, FLA1 assessors requested more SLAs than did FLA2 assessors (29 [18.6%] vs. 17 [10.9%], respectively). Three adverse events were recorded. In two the assessors differed, with one recording an adverse event and the other recording an area of concern. Additionally, in one instance the assessors differed on the preventability of the event but agreed that it may have contributed to death.

Conclusion: Concordance between assessors ranged between 'moderate' (requests for an SLA) through to 'almost perfect' (presence of an adverse event). This suggests the FLA screening process is reliable.