Causality extraction from medical text using Large Language Models (LLMs)

This section summarizes the prior work related to information extraction on the Clinical Practice Guidelines. Extracting clinical findings from notes of outpatient progress was early done realized by (Ertle et al., 1996). Fifteen years later (Taboada et al., 2013) targeted the automated extraction of diagnosis and treatment procedures from clinical guidelines. A similar work (Kaiser and Miksch, 2010) introduced a method for automatically collecting useful information using rules rooted in both syntactic and semantic information. A pattern-based approach was further used by (Chunhua et al., 2014), which contrasted a manually developed ontology for CPG eligibility criteria with a top-level ontology stemming from a semantic pattern-based approach. A more recent work (Fazlic et al., 2019) introduced an innovative system that blends together the methodologies of Natural Language Processing (NLP) and Fuzzy Logic. A supervised machine learning methodology was used by another similar work (Graham et al., 2022) to extract and categorize Conflicts Of Interest (COIs) from disclosure statements indexed in PubMed.

Recently, Large Language Models (LLMs) have (also) been employed for a variety of NLP tasks, including those involving information extraction. LLMs can be fine-tuned to cater to a specific dataset, or a prompt-based approach can be utilized. An illustrative study (Zhao et al., 2021) measures the efficiency of the few-shot learning performance of GPT-3 in tasks related to text classification and information extraction. A recent survey article (Landolsi et al., 2023) provides a summary of the methods and solutions employed for information extraction. It also highlights the challenges encountered when extracting information from medical documents (such as, ambiguities when the named entity belongs to more than one class, phrase boundary detection, name variations, and others).

The study by (Li et al., 2021) directly extracts cause and effect from text without separately extracting candidate pairs and their relations. A work on event extraction (Man et al., 2022) focuses on identifying the causal relationship between pairs of event mentions, also known as ’Event Causality Identification’ (ECI). Balashankar et al. (Balashankar et al., 2019) propose an event extraction (modality) that seeks to uncover the hidden relationships between events mentioned in news streams by creating a Predictive Causal Graph (PCG). Prompt tuning has been proposed to bridge the gap between pre-training and fine-tuning on many of the mainstream NLP tasks like text classification(Schick and Schütze, 2020; Zhang et al., 2021), information extraction(Chen et al., 2021; Cui et al., 2021) etc. In (Liu et al., 2023), Knowledge Enhanced Prompt Tuning (KEPT) employs external knowledge sourced from knowledge bases (KBs) to fine-tune pre-trained language models through the design of an attention mechanism.

A recent article (Chan et al., 2023) describes the use of ChatGPT to extract cause/effect relationships from text on three datasets: (1) Choice of Plausible Alternatives (COPA) (Gordon et al., 2012), which is a collection of premises, along with two questions related to each premise, that requires causal reasoning in order to solve the inference; (2) e-CARE (Du et al., 2022) which is an explainable causal reasoning dataset with cause, effect and two possible explanations; (3) Headline Cause (Gusev and Tikhonov, 2021) dataset, which aims to identify the implicit causal relations between pair of text. On COPA, ChatGPT in-context learning got a 97% accuracy (performance); on the eCARE dataset, a 79.6% accuracy was obtained using prompt engineering, and 72.7% accuracy was recorded for on Headline Cause.

In 2013 the task of automatic detection conditional statements in medical guidelines was first introduced (Wenzina and Kaiser, 2013). The article used a rule-based approach, focusing on presence of connectives such as ”if”, and a collection of word-based syntactic patterns. Subsequent works on detecting condition action statements from CPGs, (Hematialam and Zadrozny, 2017) and (Hematialam, 2021), apply supervised machine learning techniques to classify sentences according to whether they express conditions and actions. Another study (Hussain et al., 2018) used heuristic patterns to identify recommendation statements in Clinical Practice Guidelines (CPG). A review article (Fu et al., 2020) documents the existing methods and tools for clinical concept extraction. The summarization of biomedical literature is addressed in (Xie et al., 2022) using pre-trained language models. A more recent study (Tang et al., 2023) explores the use of ChatGPT for clinical text mining, specifically for extracting structured data from unstructured healthcare texts and focusing on biological named entity recognition and relation extraction by identifying and extracting medical entities from text related to disease and drug, symptoms and treatment, etc.

Due to the intricacy of causality extraction, which involves annotators labeling varying text spans as ”cause,” ”effect,” and so on, computing agreement between two annotators can be challenging as it requires comparing two spans of texts. Traditional methods of inter-annotator agreement, such as the Kappa statistic (Fleiss et al., 2013), are inadequate due to their need for classifications to fit into mutually exclusive and discrete categories. Therefore, we decided to assess agreement using both exact match and relaxed match criteria. The F-measure is used for the exact match (Hripcsak and Rothschild, 2005; Thompson et al., 2009; Mihăilă et al., 2013) between the labels. In the case of the relaxed match, the average distance between phrases is computed. Initially, the annotated phrases, their corresponding labels, and the full sentence they are derived from are extracted from the entire annotated document. These annotations, originating from both annotators, are then compared and amalgamated based on the sentence. The resulting merged table thus features the sentence, the extracted phrase, and the labels as marked by Annotator 1 and Annotator 2. In total, 514 matching phrases have been identified. An overall agreement computed as a Jaccard similarity of 0.66 was obtained. Details of the inter-annotator agreement computation are given below.

From the merged data table, the inter-annotator agreement was computed. This is done by computing the match between the annotations as follows.

• •

  Relaxed match – Both annotator’s phrases overlap with each other but are not necessarily an exact match.

• •

  Exact match – Both annotator’s phrases exactly match.

To execute the relaxed match, we employed the Levenshtein distance (Miller et al., 2009) and the Jaccard distance (Real and Vargas, 1996). The Levenshtein distance quantifies the difference between two string sequences, indicating the minimum single-character edits required to transform one word into another. Jaccard similarity computes the degree of relatedness between two finite samples by dividing the intersection’s size by the size of the sample sets’ union. The Jaccard distance is subsequently calculated by subtracting the Jaccard similarity from 1. The Python library Levenshtein ¹¹1https://pypi.org/project/python-Levenshtein/ is used in computing the Levenshtein distance. The Jaccard index was computed using the Python library textdistance²²2https://pypi.org/project/textdistance/. The Levenshtein distance and the Jaccard distance between the annotators are summarized in Table 1

From Table 1, we can understand that there is an average Levenshtein distance of 0.41 and an average Jaccard distance of 0.34. In most cases, both annotators annotated the same sentence with the same labels, but the length of the phrase was different. The exact match between the phrases is computed by finding the exact string match between phrases 1 and 2. Out of the 514 phrases, 112 phrases are exact matches. The match between the labels for the same phrase by both annotators is also computed with an average F1 score of 0.78. The match between the labels for each subcategory is given in Table 2

Seven documents on gestational diabetes guidelines provided by different societies are downloaded as PDF documents. The PDFs are converted into a document format, and the documents are given to the annotators for annotating them manually. The annotators used tags to annotate the documents.

After annotating them, the NLTK sentence tokenizer is used to extract sentences from all the documents. The sentences from all the documents are appended together and converted into a data frame. Regular expressions are used to extract the causal sentence. If any of the sentences contain a tag ¡¿, it will be extracted as a causal sentence. Again regular expressions are used to extract the phrases of cause, effect, action, signal, and condition from the sentences. The extracted phrases are used for computing inter-annotator agreement.

Given the good performance of DistilBERT with organizational data (Gopalakrishnan et al., 2023), this model was also applied to the medical data. Considering the limited sample size in medical data, we attempted to improve the learning process by increasing the number of epochs. This approach allows for more refined fine-tuning of the model.

In order to decide on the correct number of epochs and to avoid overfitting, we tried running the model for 100 epochs and plotted the validation loss and the training loss. The graph showing the train and validation loss for our highest performing model, BioBERT, is given in Figure 2.

From the graph, we can understand that with the increase in the number of epochs, the training loss is constantly increasing and approaching 0. The validation loss decreases till 18 epochs and then starts to increase. Based on this, we fine-tuned DistilBERT for 18 epochs, BERT(BERT-base-uncased) for 20 epochs, and BioBERT for 16 epochs.

The data is split into train and test. DistilBERT for token classification is fine-tuned on the training data for 18 epochs. On the test data, the model obtained an average F1-score of 0.57. Similarly, we fine-tuned BioBERT for 16 epochs and BERT for 20 epochs. Out of these three models, BioBERT(Lee et al., 2020) gave us an average higher F1-score. BioBERT gave an average F1 score of 0.61, and BERT gave an average F1 score of 0.60. The detailed results of fine-tuning BioBERT on the test data are given in Table 3; and, for comparison, the summary of the results of using variants of BERT for causality extraction task is given in Table 4

Generative Pre-trained Transformer 4 (GPT-4) (OpenAI, 2023) outperforms most of the state-of-the-art performing models on the traditional NLP benchmark datasets. In this section, we discuss our results of prompting GPT-4-0314 with a with context window of 8,192, for the causality extraction task. We explored various prompt sizes (zero, four, six, eight, ten-shot, and twenty-shot prompting).

As an initial step, we tried the sentence with token-level labels for each word in the sentence as prompt examples. For the test data, the model is expected to predict a label for each word in the sentence. However, the model hallucinated by predicting a longer number of labels than in the given sentence; that is, a long sequences of non-existing ”non-causal” labels.

Since GPT-4 hallucinated for the token-level predictions, we tried extracting the phrases of cause/effect relationships in text and tried converting them into token level by assigning labels for each token. We started with a four-shot prompting. The annotated data with the tags will be given as an example in the prompt, and the model is expected to predict similarly. A sample is given in Example 4.1.

We tried converting the predictions with the tags into a token-level format in order to compute the F1 score. However, since the tags are placed in different places in some of the gold annotations and predictions, the number of tokens in gold and predictions doesn’t match. An example is given below

In Example 4.2, the phrases marked indicate the scenario where some extra spaces can be added, leading to the indifference in the number of tokens between gold and the predictions. In the gold data, neonatal hypoglycemia</E> .3,9-11 have a space after the tag, but in the prediction, the tag is predicted after
the number, which leads to no space between </E> and .3,9-11. In some scenarios, the GPT-4 omits some of the words if they do not contain a causal relation (omits the ’O’ labels in some places). This mismatch between the gold and the predictions impedes the token-level comparison and reporting of the F1 score. An example is given below:

In Example 4.3, in the prediction, the keyword ”Hemoglobin” is missing, which is present in the gold data. In some places, such inconsistencies lead to token mismatch between the gold and predicted data.

To compare the performance of GPT-4 with other models, the predictions are converted into the token level and manually checked to convert both the gold predictions to the same number of tokens for the four-shot prompting. In the predictions, some tokens are missed; those tokens are added to the predictions and marked as label ”O.”(as O indicates tokens that are not cause, effect, condition, action, or signal). After converting the data into a token level, we computed the F1 score. With GPT-4, we got an average F1 score of 0.39 with four-shot prompting.

LLAMA2(Touvron et al., 2023) is a pre-trained and fine-tuned Large Language Model. Three variants of LLAMA2 are available, which differ in the parameters. 7B, 13B, and 70B parameters are publicly available. LLAMA2 is trained on two trillion tokens of data. In our experiments, the LLAMA2 7B parameter is fine-tuned on the medical data. It is fine-tuned using the HuggingFace autotrain.

To fine-tune LLAMA2, the first step is to prepare the data. At first, when the model was fine-tuned and tested on the token level as BERT, LLAMA2 was predicting a long number of ”O-other” as GPT-4. So we dealt with this as a phrase-level extraction problem. The data is prepared with three parts which are instruction, input, and output. A sample training data is given in example 5.1

The test data should be similar to the training data except for the output, which should be empty. The gestational diabetes annotated data was split into train and test data. The HuggingFace autotrain ⁵⁵5https://huggingface.co/docs/autotrain/llm_finetuning for the LLM fine-tuning was used to fine-tune the model. The fine-tuned weights are pushed into the HuggingFace dataset for inference. This experiment was done using Google Colab Pro+ with a High-RAM A100 GPU. Similar to the GPT-4, the predictions of LLAMA-2 were also at phrase level. So a similar evaluation strategy is followed for LLAMA2. We present the results with three types of distance.

The predictions are split into phrase levels and then compared with gold data. The Jaccard similarity was computed using the textdistance⁶⁶6https://pypi.org/project/textdistance/ Python library. The cosine similarity is obtained by computing the vectors of both the gold and the predictions using the Universal sentence encoder(Cer et al., 2018). The computed values are used to compute the pairwise cosine similarity between two vectors using Scikit-learn ⁷⁷7https://scikit-learn.org/stable/modules/generated/sklearn.metrics.pairwise.cosine_similarity.html.

Initially, we split the data into train and test using the Scikit learn train_test_split(). We have converted the phrase-level predictions into token-level. In the test data, there were a total of 59 samples. Out of the 59 samples, only 29 samples, LLAMA2 predicted the labels, so the evaluation is only for those sentences. With LLAMA2, we got an average F1-score of 0.36, which is lower than that of all the other models.

Since the test data size is very small, we have also tried a four-fold cross-validation on this data. The results of fine-tuning LLAMA2 using four-fold cross-validation with 3,5, and 10 epochs are given in Table 7.

With the increase in the number of epochs, both the Jaccard similarity and F1-score increase. Also, the predictions of LLAMA2 missed labels in many of the predictions. It extracted the phrases with no label. With three epochs, LLAMA2 missed 38% of the labels; with five epochs, 21% of the labels; and with ten epochs, it missed 26% of the labels. We omitted the predictions with no labels (108 predictions, 60 predictions, 76 predictions). The results of causality extraction presented in Table 7 are after omitting the predictions with no labels.

From the results, we can understand that Jaccard similarity, cosine similarity, and F1 score increase with the increase in the number of epochs. However, the number of missed labels started increasing after 5 epochs.