Large Language Models are good medical coders,
if provided with tools

Medical coding is a critical process in healthcare systems, essential for accurate billing, epidemiological studies, and healthcare quality assessment [1, 2]. The recent paper “Large Language Models Are Poor Medical Coders — Benchmarking of Medical Code Querying” published in NEJM AI [3] highlighted significant limitations in the ability of large language models (LLMs) to accurately generate medical codes.

The application of artificial intelligence (AI) and machine learning (ML) in healthcare, particularly in clinical coding, has been a subject of increasing interest in recent years [4, 5]. Previous studies have explored various approaches to automate medical coding, including rule-based systems [6], traditional machine learning methods [7], and more recently, deep learning techniques [8].

Soroush and colleagues evaluated the performance of several prominent LLMs, including GPT-3.5, GPT-4, Gemini Pro, and Llama2-70b Chat, in querying medical billing codes. Their study encompassed a comprehensive dataset of ICD-9-CM, ICD-10-CM, and CPT codes extracted from the Mount Sinai Health System electronic health record. The authors found that even the best-performing model, GPT-4, achieved exact match rates of only 45.9% for ICD-9-CM, 33.9% for ICD-10-CM, and 49.8% for CPT codes. These results led to the conclusion that LLMs are currently not suitable for direct use in medical coding tasks.

However, we hypothesized that the performance of LLMs in medical coding could be significantly improved by providing them with appropriate tools and retrieval mechanisms. This approach aligns with recent advancements in retrieval-augmented generation [9] and the use of external knowledge bases to enhance LLM performance [10].

To test this hypothesis, we designed an experiment using a combination of the Colbert-V2 retriever [11] and GPT-3.5-turbo for reranking. Our approach aimed to address the limitations observed in the direct code generation method used in the NEJM study, drawing inspiration from successful applications of similar techniques in other domains [12].

In this paper, we present our methodology and results, which demonstrate a substantial improvement in medical coding accuracy. By achieving a 100% exact match rate on a sample of 100 codes, our findings suggest that LLMs, when equipped with the right tools, can indeed be effective in medical coding tasks. This study not only challenges the conclusions of the NEJM paper but also opens new avenues for the application of AI in healthcare information management, potentially addressing long-standing challenges in medical coding efficiency and accuracy [2, 13].

Figure 1 illustrates the workflow and results of our proposed Retrieve-Rank system compared to the control group. This visual representation highlights the significant improvement in accuracy achieved by our approach.

The following sections will detail our experimental setup, results, and discuss the implications of our findings for the future of automated medical coding.

Our study employs a methodology similar to that used in the "Large Language Models Are Poor Medical Coders — Benchmarking of Medical Code Querying" paper, utilizing single-term medical conditions as inputs for ICD-10-CM code prediction. However, we introduce a novel two-stage Retrieve-Rank system, inspired by Doosterlinck’s Infer-Retrieve-Rank framework[14], which significantly improves upon previous approaches.

Our approach involves the following steps:

1. 1.

   Retrieval: Given a single-term medical condition, we use ColBERT-V2 to retrieve the top-k (k=15) most relevant ICD-10-CM codes from our trained index.

2. 2.

   Reranking: We use GPT-3.5-turbo to rerank the retrieved codes and select the most likely ICD-10-CM code for the given condition.

We utilized Ragatouille to train and develop a ColBERT-V2 RAG (Retrieval-Augmented Generation) system based on ICD-10-CM data downloaded from the CDC website¹¹1https://www.cdc.gov/nchs/icd/icd-10-cm/files.html.

It’s important to note that while we use simplified single-term inputs similar to previous studies, our two-stage approach allows for more nuanced and accurate code prediction. This methodology, while not fully representative of the complexity found in real-world medical coding scenarios, allows for a direct comparison with previous findings and provides insights into the improved capabilities of our Retrieve-Rank system.

We designed an experiment to evaluate the performance of our ColBERT-V2 RAG system against a control group. The experiment was implemented using Python, with the following key components:

• •

  Data Preparation: We used a CSV file containing single-term medical conditions and their corresponding ICD-10-CM codes.

• •

  Sampling: The experiment randomly sampled 100 entries from the dataset.

• •

  Code Normalization: ICD-10-CM codes were normalized by removing periods and converting to uppercase to ensure consistent comparison.

• •

  Prediction: For each sampled entry, we used our RAG system to predict the ICD-10-CM code based on the single-term medical condition.

• •

  Evaluation Metrics: We focused on the top-one accuracy, comparing the predicted code with the true code. A match was considered successful if the main part of the predicted code (before any subdivisions) matched the true code.

• •

  Control Group: We implemented a control group using GPT-3.5-turbo to provide a baseline for comparison. This model was prompted with "You are a medical coding expert that can suggest an ICD-10-CM code for a given query." followed by the single-term medical condition.

• •

  Results Logging: The experiment results, including the conditions, true codes, predicted codes, and match results, were logged to a CSV file for further analysis.

This experimental setup allowed us to directly compare the performance of our ColBERT-V2 RAG system against a simpler baseline model, providing insights into the effectiveness of our approach for ICD-10-CM code prediction, even with simplified inputs.

While our results show significant improvement over the Vanilla LLM approach, we acknowledge that further research using more complex, realistic medical cases is necessary to fully evaluate the potential of the Retrieve-Rank system in practical applications.

We evaluated our two-stage Retrieve-Rank system against a Vanilla LLM using GPT-3.5-turbo on a dataset of 100 diagnosis description with corresponding ICD-10-CM codes. The results demonstrate a significant performance improvement over the baseline method.

Our study demonstrates the significant potential of the two-stage Retrieve-Rank system in automating ICD-10-CM medical coding. The system’s perfect accuracy across a diverse set of 100 diagnosis descriptions, compared to the 6% accuracy of a Vanilla LLM, underscores the effectiveness of combining retrieval and ranking mechanisms in tackling complex coding tasks.

The Retrieve-Rank system exhibited remarkable precision in capturing crucial details such as specific anatomical locations, encounter types, and intricate medical conditions. Its consistency across various medical specialties further highlights its versatility and potential for broad application in healthcare settings.

While these results are encouraging, we acknowledge the limitations of our study, particularly the relatively small sample size. Future research should focus on validating these findings with larger, more diverse datasets and exploring the system’s performance on edge cases and rare conditions.

The implications of this research are significant for the healthcare industry. An accurate, automated coding system could substantially reduce the workload on medical coders, minimize coding errors, and improve the overall quality of medical records. This, in turn, could lead to more efficient healthcare administration, more accurate billing processes, and potentially better patient care through improved data quality for medical research and decision-making.

As we move forward, it will be crucial to continue refining and testing the Retrieve-Rank system, possibly incorporating advances in language models and retrieval techniques. Additionally, exploring ways to make the system interpretable and adaptable to evolving medical knowledge will be key to its practical implementation in healthcare settings.

In conclusion, while further research is needed, our study presents a promising step towards more efficient and accurate automated medical coding, contributing to the ongoing digital transformation of healthcare administration.

The complete dataset of 100 medical cases, including predictions from both systems, is available as an ancillary file with this arXiv submission. Additional materials, including detailed methodology and error analysis, are also provided as ancillary files.

The code used to conduct the experiments and analyze the results is publicly available on GitHub at https://github.com/ainativehealth/GoodMedicalCoder. This repository contains:

• •

  Python scripts for running the ICD-10 code prediction experiment (experiment.py)

• •

  Code for creating the index using the RAG model (index.py)

• •

  ICD-10 code datasets (ICD-10.csv and ICD-10_formatted.csv)

• •

  Requirements file listing all necessary Python dependencies (requirements.txt)

• •

  Detailed instructions for reproducing the experiments

Researchers interested in replicating or building upon this work can access all necessary code and data through this GitHub repository. The repository is open-source and licensed under the Apache-2.0 license, allowing for broad use and adaptation of the materials.