A Comparative Study on Automatic Coding of Medical Letters with Explainability

Most papers regarding general healthcare NLP can be divided into two topics: text classification and information extraction Dong et al. (2022). Classification can be split into three versions, each getting more complex: binary classification, where an instance is in one of two distinct categories (e.g., smoker or non-smoker); multi-class classification, where there are multiple categories, but an instance can still only be assigned to one class (e.g., current smoker, former smoker, non-smoker); and multi-label text classification ⁶⁶6https://huggingface.co/blog/Valerii-Knowledgator/multi-label-classification. This involves instances that can be associated with several different labels/categories simultaneously, such as discharge letters, in which each letter always contains multiple conditions. Automated medical coding is often identified as a multi-label text classification problem; however, some older attempts still utilise information extraction or a combination of methods from both topics. The first attempts at automated clinical coding were from around 1970, such as this 1973 study by Dinwoodie and Howell (1973) that utilises a ‘fruit machine’ methodology. This entails representing each significant word of a diagnosis with an associated code number and, like a fruit machine in a pub, the code is correct when a common code number appears for all words in the diagnosis. While this study returns impressive results with a correct coding rate of over 95%, this is only done with a small collection of pre-coded morbidity data from 16 doctors around Scotland. Thus, the project will not scale up to the complex real-world scenario.

No real progress was then made for the next few decades. A 2010 literature review on clinical coding Stanfill et al. (2010) evaluated the results of 113 studies, the earliest being the above 1973 study, and concluded that while the systems hold promise, there has been no clear trend of improvement over time. Another interesting trend from this review is that, while no improvements had been made, researchers’ interest was increasing, as all but 4 of the studies found were published after 1994. Examples of attempted innovation from this period include a study from Farkas and Szarvas (2008) focusing on rule-based automated radiology report coding. It uses a variation of multi-label classification that treats the assignment of each label as a separate task, as opposed to treating valid sets of labels as a single class. It then builds a rule-based expert system that operates on if-then codes through the ICD hierarchy. It uses decision trees (which recursively classify the data through conditions, similar to the rule-based system used to classify codes) to predict false positives, which occur when the model incorrectly predicts a positive outcome. It then uses a maximum entropy classifier to tackle false negatives, calculating each token’s probability of a false negative. Both the decision tree and max entropy classifier worked to increase the micro-averaged $F_{\beta=1}$ scores by   4%, to 87.92%.

While these rule-based solutions are very accurate for the specific types of documents they examine, they will not generalise well to new problems since they are domain-specific. For them to be feasible for real-world use, the rules would need to be extended to tens of thousands of codes and would require a substantial investment of time and expertise to be executed properly. Statistical approaches such as initial attempts from Mullenbach et al. (2018), which utilised logistic regression (LR), and Perotte et al. (2014), which made use of Support Vector Machines (SVM), were attempted. However, the results on the full MIMIC database (shown in Figure 1) indected that they were also infeasible. Therefore, a different method had to be attempted: deep learning and neural networks.

The general approach of deep learning in neural networks aims to map a complex function learned through the training data to match the information in the text to an appropriate set of medical codes Dong et al. (2022). Before any deep learning is completed, the common first step in these projects - aside from preprocessing - is to produce word embeddings for each token. Each embedding is a semantically meaningful mathematical representation, usually a vector, of the token designed so that tokens with similar meanings have similar vectors Percha (2021). To compare the meaning of two words, one calculates the cosine similarity of their corresponding vectors. The most common method for doing this is ‘word2vec’, which operates on the assumption that words with similar meanings tend to occur in similar contexts. It uses either a continuous bag of words (CBOW) model that predicts the target words based on the context words (words surrounding the target word) or a skip-gram that predicts the context words based on the target words Mikolov et al. (2013), both of which are examples of single-layer neural networks. A more advanced version of word2vec that strays from the standard embedding practice of one vector per word/token/document, is the development of bidirectional encoder representations from transformers (BERT) Devlin et al. (2019). These are massive pre-trained language models that are too resource-intensive to be trained from scratch in most circumstances, however, models trained on a general corpus can be fine-tuned to meet specific needs (such as clinical text mining through transfer learning Peng et al. (2019)). Unfortunately, due to their size and complexity, they are not currently feasible to be trained on larger datasets without significant modification. The first successful deep learning attempts utilised recurrent neural networks (RNNs), with a focus on two specific types: Gated Recurrent Units (GRUs) and Long Short-Term Memory Networks (LSTMs). The project Nigam (2016) constructs an RNN with a single layer consisting of 20 time steps; with each time step, a normalised vector representing a patient note is submitted in a time sequential order (oldest to most recent). The activation (threshold) function is tanh, a mathematical operation applied to the weighted sum of inputs and biases in each neuron that introduces non-linearity into the network. There is a dropout rate of 0.1 that is applied to prevent overfitting during training, and a learning rate of 0.001 is used to determine how much the weights of the network are updated during each training iteration. Finally, the model uses cross-entropy loss as its sigmoid function, normalising the neuron’s output to a value between 0 and 1.

GRUs are implemented as recurrent units, where each unit contains a reset gate and an update gate, which allow the GRU to regulate the flow of information and selectively update its hidden state. They are computationally more efficient; however, they may be outperformed by LSTMs in tasks requiring long-range dependencies. LSTMs are built like GRUs but using three gates instead of two: an input gate, a forget gate, and an output gate. They are more powerful due to their additional gates and memory cells that allow them to better preserve information over time. Convolutional neural networks (CNNs) consist of convolution layers and pooling layers and are mainly used for image and video processing, however, if the text is manipulated and processed correctly, they can be very effective for text processing. For example, one of the most successful studies into automated medical coding is the 2018 project Convolutional Attention for Multi-Label Classification (CAML) Mullenbach et al. (2018), which utilised a CNN but swapped the pooling layer for an attention mechanism. This attention mechanism is applied to the data to identify relevant portions of the document for each code prediction, allowing it to selectively focus on and assign higher importance to the relevant words and phrases Vaswani et al. (2017). Using attention mechanisms in this way also allows for enhanced interpretability. It provides insights into which parts of the document it made its predictions from, instead of just being put through a function as with previous methods.

With attention comes transformer-based networks, and while attention networks are not exclusively transformer-based, transformers are exclusively attention-based Vaswani et al. (2017). They rely solely on self-attention mechanisms, parallel processing the entire input sequence. This makes them more efficient for handling long sequences and allows for faster training and inference than more sequential models like RNNs. Transformers also allow for multi-head attention, an extension of the self-attention mechanism that allows the model to further parallelize the processing, enabling transformers to capture different aspects of the input data in parallel, allowing for more complex modelling of the relationships and patterns. This has recently been introduced into automated medical coding and, as demonstrated with HiLAT Liu et al. (2022), it is already promising. However, due to the computational complexity of such a model, it has only been tested on the limited MIMIC-III-50 dataset.

The table shown in Figure 1 demonstrates automated coding techniques’ slow but consistent progress. The highlighted segments represent the top performers in their respective categories. The transformer-based HiLAT model outperforms every other model in every metric when tested on the MIMIC-III-50 database. On the other hand, the CNN + attention-based model of CAML does the same when tested against all the models on the MIMIC-III Full database, while it is also the only model that can provide a level of explainability to its answers. These results indicate that an attention-based model is the preferred choice due to the superior results and their ability to provide explainability for their answers.

In Clinical NLP, the first resource is the MIMIC-III dataset, which is the only publicly available mainstream English dataset with enough data to perform proper training. Additionally, most models that attempt to solve the automatic coding problem use this dataset.

MIMIC-III Johnson et al. (2016) is a large, freely available database comprising de-identified health-related data associated with over forty thousand patients who stayed in critical care units of the Beth Israel Deaconess Medical Centre between 2001 – 2012 ⁷⁷7https://mimic.mit.edu/docs/iii/. The database is freely available to researchers worldwide, provided they have become a credentialed user of PhysioNet Johnson et al. (2016) and completed the required ‘Data or Specimens Only Research’ CITI training ⁸⁸8https://physionet.org/content/mimiciii/view-required-training/1.4/ (Or another recognized course in protecting human research participants that includes HIPAA requirements). All data in the MIMIC database has been deidentified per HIPAA (Health Insurance Portability and Accountability Act) standards. This ensures that all 18 listed identifying data elements, such as names, telephone numbers, and addresses, are removed. The only thing not removed are dates, which are shifted in a random but consistent manner to preserve intervals. Therefore, all dates occur between 2100-2200, but the time of day, day of the week, and approximate seasonality have been conserved.

MIMIC is a relational database consisting of 26 tables containing different forms of data, from the patient’s clinical notes in NOTEEVENTS to extremely granular data such as the hourly documentation of patients’ heart rates. This makes it a vast and complex database to work with - however since we are only using the database for its clinical notes, only five tables are required:

• •

  NOTEEVENTS – Deidentified notes, including nursing and physician notes, ECG reports, imaging reports, and discharge summaries.

• •

  DIAGNOSES_ICD - Hospital-assigned diagnoses, coded using the International Statistical Classification of Diseases and Related Health Problems (ICD) system.

• •

  PROCEDURES_ICD - Patient procedures, coded using the International Statistical Classification of Diseases and Related Health Problems (ICD) system.

• •

  D_ICD_DIAGNOSES - Dictionary of International Statistical Classification of Diseases and Related Health Problems (ICD) codes relating to diagnoses.

• •

  D_ICD_PROCEDURES - Dictionary of International Statistical Classification of Diseases and Related Health Problems (ICD) codes relating to procedures.

This still leaves a lot of unnecessary data. For example, the NOTEEVENTS table contains CHARTTIME, CHARTDATE, and STORETIME, which are the time and date a note was charted and the time it was stored in the system. The notes in NOTEEVENTS vary in usefulness and format, with the type of note indicated in the DESCRIPTION column. Since all the medical coding projects that use MIMIC unanimously choose to use the discharge summaries as they contain the most potential codes per letter (15.9 labels per document). We removed all the other types of notes. This was done by creating a new table that copied each line as long as the DESCRIPTION = ‘Discharge Summary’. The next step is to combine the data in separate tables into one table for easier access.

Another note on MIMIC is about its most popular subset, MIMIC-III-50, that contains only the notes and codes of the top 50 most frequently occurring codes (Table 1). First occurring in CAML Mullenbach et al. (2018), MIMIIC-III-50 is often used as a proof-of-concept database for automatic medical coding projects due to it being significantly smaller (8,067 documents compared to 47,724) and with fewer labels (5.7 compared to 15.9 for MIMIC full), which means it takes less time and computational resources to train against. Projects like HiLAT Liu et al. (2022) that face challenges in accessing the necessary computing power for training their models have utilised the MIMIC-III-50 dataset to train on and achieve state-of-the-art results. The only issue with using MIMIC-III-50 is that, as Figure 2 demonstrates, it doesn’t give the same opportunity to test models against a long tail distribution.

A database that follows a long tail distribution is one where there are many data points that are not well-represented, and the majority of occurrences are concentrated around a few values at the “head” of the distribution Zhang et al. (2023). This accurately describes the MIMIC-III-Full database, where the top 105 codes make up 50% of the total labels in the set, and there are 3,110 labels that have fewer than 5 examples Nigam (2016), with 203 codes not appearing in any discharge summaries at all. Solving the long tail distribution of MIMIC is one of the key challenges that will need to be addressed by the potential models to be deployed.

Before each selected model is evaluated, the problem needs to be formally defined. Taking $\mathcal{X}$ as the collection of clinical notes and $\mathcal{Y}$ as the full set of labels (ICD-9 codes). Each instance $x_{d}\in X$ is a word sequence of a document, $d$, and is associated with label set $y_{d}\subseteq Y$, where each $y_{d}$ can be represented as a $|Y|$ multi hot vector (a vector where multiple elements can have a value of 1, indicating multiple features/categories are present at the same time), $\overrightarrow{Yd}=[y_{d1},y_{d2},...,y_{d|Y|}]$, and $y_{dl}\in(0,1)$ where $l$ indicates the $l\textquoteright th$ label has been used for the $dth$ instance and 0 indicates irrelevance Perotte et al. (2014). From this, the task of the models is to learn a complex function $f:\mathcal{X}\rightarrow\mathcal{Y}$ from the training set.

All the chosen models use the same loss function, binary cross-entropy, and optimise it with L2 normalisation using the Adam (Adaptive Movement Estimation) optimiser Kingma and Ba (2014). Loss functions are used in neural networks as a measure of how well the networks predictions match the true values of the training data, with binary cross entropy loss measuring the dissimilarity between the true binary labels and the predicted probability of the model. In the context of these models, L2 normalisation is used to avoid overfitting, which occurs when the model is trained so well on a particular dataset that it fails to generalise well to new, unseen data. To prevent this, penalty terms proportional to the magnitude of the vectors (Euclidean norm) are added, which penalise overly specific mappings and encourage the model to learn simpler, more generalised weight configurations. The Adam optimiser is a popular optimisation algorithm used to update the parameters of a neural network to minimise the loss function during training.

CAML Mullenbach et al. (2018) (Figure 4), as already mentioned in the background section, utilises a CNN based architecture but swaps the traditional pooling layer for an attention mechanism. The model starts by horizontally concatenating pretrained word embeddings into a matrix, X. A sliding window approach as is standard in CNNs is then applied to this matrix that computes an equation on each section of the matrix, resulting in the matrix H.

Next, the model applies a per-label attention mechanism. For each label, $l$, the matrix vector product is computed, and the result of this is passed through a SoftMax operator that essentially reduces the input values to the range [0,1] while ensuring that they sum up to 1 so they can be used as probabilities. This SoftMax operator returns the distribution over locations in the document in the form of attention vector $\alpha$. This attention vector is then used to compute vector representations for each label, $vl$. Finally, a probability is computed for label $l$ using another linear layer and sigmoid transformation to obtain the final label predictions $yl$. This normalisation process ensures that the probability of the label is normalised independently rather than normalising the probability distribution over all labels like the SoftMax operator does.

The HLAN model Dong et al. (2021) is built around providing explainability for its results, and consists of an embedding layer, the HLAN layers, and a prediction layer. The embedding layer converts each token in the sentence into a continuous vector where the word embedding algorithm word2vec returns the vector of word embeddings $x_{di}$.

The HLAN makes extended use of Gated Recurrent Units (GRU) to capture long-term dependencies. The GRU unit processes tokens one by one, generating a new hidden state for each token. At each hidden state, the GRU considers the previous tokens using a reset gate and an update gate. The GRU method implemented is known as Bi-GRU because it reads the sequence both forwards and backwards, concatenating the states at each step, to create a more complete representation.

The label wise word-level attention mechanism, which contains a context matrix ($V_{w}$) where each row $V_{wl}$, is the context vector to the corresponding label $y_{l}$. The attention score is calculated as a SoftMax function of the dot product similarity between the vector representation of the hidden layers from the Bi-GRU and the context vector for the same label. The sentence representation matrix $C_{s}$ is computed as the weighted average of all hidden state vectors $h^{i}$ for the label $y^{i}$.

The label-wise sentence-level attention mechanism is computed in much the same way, outputting sentence-level attention scores and the document representation matrix $C_{d}$. The prediction layer then utilises a label-wise, dot product projection with logistic sigmoid activation to model the probabilities of each label to each document. Finally, the binary cross entropy loss function is optimised with L2 normalisation and the Adam optimiser.

The HLAN has an extra label embedding initialisation (denoted as +LE) that can be implemented in place of the normal embedding layer and functions by leveraging the complex semantic relations (how different elements are related to each other in terms of their meanings) among the ICD codes. The embedding works off for two correlated labels; one would expect the prediction of one label to impact the other for some notes, which is represented as giving each label representation corresponding weights. The HLAN model was based on the HAN model Yang et al. (2016), where the only difference between the two is that at the sentence and document level, HLAN utilises contextual matrices, whereas HAN uses contextual vectors. This means that while HLAN is more individually label-oriented, HAN still produces an attention visualisation for the whole document and the results are only slightly worse but reducing the computational complexity of training the model. HAN Yang et al. (2016) model was originally proposed as “Hierarchical Attention Networks for Document Classification”.

Much like HLAN, MHLAT Duan et al. (2023) is comprised of three main components: an input/encoder layer, MHLAT layer, and a decoder layer (Figure 6). It also utilises the same label-wise attention mechanism, however, that is where the similarities end. In the encoding layer, MHLAT first splits the text into chunks with 512 tokens per chunk. It then adopts the general domain pre-trained XLNet Yang et al. (2019) (similar to BERT but less computationally expensive), which is further trained on MIMIC, and then applied to every chunk. Each chunk from the text is then concatenated to form a global vector of the input text, H.

While using label-wise attention through multiple passes is utilised for both HLAN and MHLAT, where HLAN uses multiple Bi-GRUs increasing the scope each time, MHLAT presents a ‘multi-hop’ approach. Initially, the label-wise attention is derived from matrices of the tokens of the input sentence from the encoder, followed by a ‘fusion’ operation that combines label-specific representations and label embeddings. A hop function is then defined that iteratively updates context information and label embeddings, which is then repeated. The decoding layer implements an independent linear layer for computing the label score and utilises the same binary cross entropy loss function as the other models.

If going purely off results (given in Figure 7), the MHLAT model returns state-of-the-art performance compared to the others in every metric it had resulted in. However, it is worth noting that the model, despite being attention-based, did not factor any type of explainability into itself. As mentioned in the motivations, we want to explore some level of interpretability of coding models, otherwise, the professionals (clinicians) using them would have no way to verify the results and build trust.

Looking at the results of the remaining models, it is clear that HLAN performs better than CAML, which in turn performs better than HAN. However, the objective of the project was to prioritise explainability in the results, which made HLAN/HAN the ideal model as despite a slight reduction in performance for the MIMIC Full dataset. The enhanced interpretability in its answers justifies its use, especially in domains such as medical coding where transparency and understanding of the models’ decisions are crucial.

The preprocessing (Figure 8) takes three of the tables from MIMIC described in Section 2.3, NOTEEVENTS, PROCEDURES_ICD, and DIAGNOSES_ICD, and combines them into one table, notes_labeled, with the schema SUBJECT_ID, HADM_ID, TEXT, LABELS where:

• •

  SUBJECT_ID – identifier unique to a patient, found in NOTEEVENTS.

• •

  HADM_ID – identifier unique to a hospital stay, found in NOTEEVENTS.

• •

  TEXT – The free text of the document. There can be multiple documents with the same HADM_ID. Found in NOTEEVENTS.

• •

  LABLES – ICD_9 labels professionally assigned and stored in sequence order in either DIAGNOSES_ICD or PROCEDURES_ICD, depending on if they were diagnoses or procedures.

This is accomplished by first concatenating both _ICD tables into one table of codes, ALL_CODES. In the next step it preprocesses the raw TEXT from NOTEVENTS, removing tokens that contain no alphabetic characters (i.e., removing 500 but not 500mg), removing white space, and lowercasing all tokens. The processed text is stored in the disch_full table, which is then joined on the HADM_ID of each line to the ALL_CODES table to form the notes_labeled table.

The code then generates the MIMIC_III_50 database by iterating through the notes_labeled file, counting the occurrences of each code, and saving the HADM_IDs to 50_hadm_ids and the codes to TOP_50_CODES. Both the standard notes_labeled and the dev_50 tables are split 90/10 to train/test respectively and stored in the train/test version of their tables.

When attempting to train the HLAN model on the full MIMIC dataset, the system that it was being trained on (our local PC) did not have sufficient memory, therefore the HAN model Yang et al. (2016) was used instead. This model did not need to be trained as the pretrained model could be downloaded from the GitHub.

There is now a working model that took a text document as input and outputted an attention visualisation in Excel and a list of predicted codes in the console.

Now with a working model, the next step is to map the ICD codes to SNOMED (Figure 9). The map ⁹⁹9https://www.nlm.nih.gov/research/umls/mapping_projects/icd9cm_to_snomedct.html was originally created for the Unified Medical Language System (UMLS) to facilitate the translation of legacy data still coded in ICD-9 to SNOMED CT codes. Therefore, it is perfect for the project’s needs. It does contain multiple columns of data that are not required, mainly usage statistics, however, these can just be ignored. The 202212 most recent release of the map was implemented by UMLS and is split up into two tab-delimited value files with the same file structure; one for one-to-one mappings, and one for one-to-many mappings. The one-to-one mapping contains 7,596 mappings (64.1% of ICD-9 codes), with each line in the file being a separate mapping. For example, the ICD code 427.31 (Atrial Fibrillation) maps directly to the SNOMED code 49436004 (Atrial Fibrillation (disorder)). The one-to-many file contains 3,495 mappings (29.5% of ICD-9 codes), with the mapping being one ICD code to multiple SNOMED codes. The file is set out as one-to-one maps, with the one ICD code being repeated for each of the many SNOMED codes, for example:

• •

  719.46 – Pain in joint, lower leg $|$ 202489000 – Tibiofibular joint pain

• •

  719.46 – Pain in joint, lower leg | 239733006 – Anterior knee pain

• •

  719.46 – Pain in joint, lower leg | 299372009 – Tenderness of knee joint

This was implemented by first loading the one-to-one map into a dictionary, then iterating through the predicted_codes list. At each iteration (new ICD code) the program checks to see if the ICD code is in the one-to-one map. If it is, the associated SNOMED code and FSN (fully specified name) are outputted; if not, the one-to-many map is loaded as a dictionary.

The program searches for the ICD code in the one-to-many dictionary, and if found, it outputs all the SNOMED codes related to the ICD code. This is done so that even if the program cannot find a direct mapping, it can at least provide the user with potential options. If an ICD code cannot be found in any mappings, the system will print the ICD code description from either D_ICD_DIAGNOSES or D_ICD_PROCEDURES. There are only a few cases, approximately 6.4% of the ICD codes, where there are no mappings available. This usually occurs with catch-all NEC (not elsewhere classified) ICD codes, such as 480.8 - Pneumonia due to other virus not elsewhere classified, for which SNOMED has no alternative mappings available.

After all these steps, the project now takes notes as input through a text document, processes them using the HAN model, and calculates the attention levels of the ICD codes. The program then converts the ICD codes into SNOMED codes with as many 1-to-1 mappings as it can find, outputting that to the console (Figure 10). Finally, the attention visualisation is exported into Excel (Figure 11) which shows each word in the file and highlights it in a shade of blue. The deeper the blue highlight, the greater the weight that word had when calculating the ICD codes. The visualisation displayed in Figure 11 is split up halfway down for ease of viewing. In reality, the left-hand side of the upper picture and the right-hand side of the lower picture are joined next to each other.

The experiments are evaluated in two ways – first, the model is tested against the standard testing scores of micro/macro F1 and precision. Second, the implementation of SNOMED mapping is also considered, calculating the percentage of codes it can predict/give options for.

To accurately test the model, data had to be gathered by running the model against MIMIC discharge summaries from the test files. This was accomplished by randomly selecting 100 notes from the test_full file (refer to sample size and model confidence by Gladkoff et al. (2022)). We then ran each set of notes through the model and put it through a program that returned the true and false positives, as well as the false negatives from the results by comparing the labels generated by the model to the true labels in the file, where:

• •

  True Positives – when the model predicts a label, and it is correct.

• •

  False Positives – when the model predicts a label, but it is incorrect.

• •

  False Negatives – when the model doesn’t predict a label even though there is a correct label.

Now that these values were generated, the model was tested against the same metrics that have been used in all the models previously.

• •

  Recall - measures how often a model correctly identifies positive instances (true positives) from all the actual positive samples in the dataset ¹⁰¹⁰10https://www.evidentlyai.com/classification-metrics/accuracy-precision-recall and is calculated by dividing the number of true positives by the number of positive instances (true positives + false negatives).

• •

  Precision – measures how often a model correctly predicts the positive class, calculated by dividing the number of correct positive predictions (true positives) by the total number of instances the model picked as positive (both true and false positives). The precision results from earlier models were with P@5, P@8, or P@15, which means measuring the proportion of relevant items within the top 5, 8, or 15 items retrieved by the system.

• •

  F1 Score – Calculated as the harmonic mean of the precision and recall scores, therefore, encouraging similar values for both precision and recall. The more the precision and recall deviate from each other, the worse the score.

• •

  Macro F1 score - is an average of the F1 scores obtained, representing the average performance of the model across all classes (each class having the same weight).

• •

  Micro F1 score - computes a global average F1 score by counting the sums of the true positives, false negatives, and false positives and then putting those into the normal F1 equation. It essentially computes the proportion of correctly classified observations out of all observations (each token having the same weight).

Aside from gathering these results, the other data collected was that of the SNOMED scores. This was gathered when running the same tests to find the other values, and each returned SNOMED score could be grouped into one of 4 categories:

• •

  1-to-1 – The ICD to SNOMED code was a one-to-one match

• •

  1-to-M – The ICD to SNOMED code was a one-to-many match

• •

  No Map – No ICD to SNOMED map was found.

• •

  No DESC – There was no description found associated with the ICD codes in the D_DIAGNOSES_ICD MIMIC file. This was a rare valid return due to the formatting of the D_DIAGNOSES_ICD file.