Assessment of Automating Safety Surveillance From Electronic Health Records: Analysis for the Quality and Safety Review System

Background and objectives: In an effort to improve and standardize the collection of adverse event data, the Agency for Healthcare Research and Quality is developing and testing a patient safety surveillance system called the Quality and Safety Review System (QSRS). Its current abstraction from medical records is through manual human coders, taking an average of 75 minutes to complete the review and abstraction tasks for one patient record. With many healthcare systems across the country adopting electronic health record (EHR) technology, there is tremendous potential for more efficient abstraction by automatically populating QSRS. In the absence of real-world testing data and models, which require a substantial investment, we provide a heuristic assessment of the feasibility of automatically populating QSRS questions from EHR data.

Methods: To provide an assessment of the automation feasibility for QSRS, we first developed a heuristic framework, the Relative Abstraction Complexity Framework, to assess relative complexity of data abstraction questions. This framework assesses the relative complexity of characteristics or features of abstraction questions that should be considered when determining the feasibility of automating QSRS. Questions are assigned a final relative complexity score (RCS) of low, medium, or high by a team of clinicians, human factors, and natural language processing researchers.

Results: One hundred thirty-four QSRS questions were coded using this framework by a team of natural language processing and clinical experts. Fifty-five questions (41%) had high RCS and would be more difficult to automate, such as "Was use of a device associated with an adverse outcome(s)?" Forty-two questions (31%) had medium RCS, such as "Were there any injuries as a result of the fall(s)?" and 37 questions (28%) had low RCS, such as "Did the patient deliver during this stay?" These results suggest that Blood and Hospital Acquired Infections-Clostridium Difficile Infection (HAI-CDI) modules would be relatively easier to automate, whereas Surgery and HAI-Surgical Site Infection would be more difficult to automate.

Conclusions: Although EHRs contain a wealth of information, abstracting information from these records is still very challenging, particularly for complex questions, such as those concerning patient adverse events. In this work, we developed a heuristic framework, which can be applied to help guide conversations around the feasibility of automating QSRS data abstraction. This framework does not aim to replace testing with real data but complement the process by providing initial guidance and direction to subject matter experts to help prioritize, which abstraction questions to test for feasibility using real data.