Applicability of Large Language Models and Generative Models for Legal Case Judgement Summarization

We try 5 different extractive summarization methods. Three of these methods – CaseSummarizer [3], BertSum [44], SummaRunner [45] – were observed to perform well for legal case judgement summarization in the prior work [46]. Additionally, we apply two recent methods PACSUM [40] and HipoRank [41].

For the IN-Abs dataset, we use the training dataset of IN-Abs of 7030 (legal document, summary) pairs to train the BertSum model. For the UK-Abs dataset, we use the training dataset of UK-Abs of 693 (legal document, summary) pairs to train the BertSum model. Since the gold standard summaries in IN-Abs and UK-Abs are abstractive in nature, we take a sentence from a gold-standard summary and compare it with all sentences in the corresponding source judgement. Based on [47], we take the 3 sentences with highest Rouge-2 F1 from the main judgement to be used as a part of the reference pseudo-extractive summary for training the model. We perform this step for every sentence in the abstractive gold-standard summary and we take a union of all the corresponding sentences obtained from the main judgement to be labelled as ’1’ for the binary classification task. All other sentences in the main judgment are labelled as ’0’ to train the BertSum model.

We try out the following Large language models (LLMs). All the LLMs take as input a ‘prompt’ and generate text as a ‘response’. Specifically for the summarization task, the prompt consists of (i) the text to be summarized, which we refer to as <text to summarize> and (ii) an ‘instruction’ that tells the model that the input text has to be summarized. Note that the LLMs have a limit on the maximum number of tokens possible in a prompt+response, as stated below.

We tried two different legal domain-specific abstractive summarization models namely Legal-Pegasus (abbreviated as LegPegasus) and Legal-LED (abbreviated as LegLED).

As stated earlier, Text-Davinci-003 and Turbo-Gpt-3.5 have a token limit of maximum 4,096 for (prompt+generated text). One token is approximately $\frac{3}{4}$ words, i.e., 1000 tokens correspond to around 750 words.⁸⁸8Detailed explanation about tokens available at https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them Also, legal domain-specific abstractive summarization models like LegPegasus have a maximum input token length of 1,024.

Recall from Section 3 that the average length of UK case judgements in the UK-Abs test dataset is 14,476 words and that of the Indian case judgements in the IN-Abs test dataset is 4,782 words. Thus, these legal case judgements are often much longer than what can be input into the summarization models/LLMs at once, and hence we have to follow a divide-and-conquer approach with the long legal documents. Our strategy involves chunking long legal documents into smaller segments or chunks of at most $K$ words (where $K$ can be 1,024 or 2,048 or higher), and each chunk is passed individually into the summarization models to obtain the output summary. Then the summaries generated for all the chunks (of a given document) are appended together in the order in which the chunks appear in the main document, to form the final output summary for the legal judgement. For legal documents with a length of lesser than 1024 words, the summary is obtained at once by passing the entire document through the summarization models.

For the ChatGPT and Davinci models, we experiment with $K$ = 1,024 and 2,048. For the chatgpt-16k model, we divide a legal judgement into longer chunks of length $K$ = 8192 words (10,922 tokens approx). Note that almost all documents in the IN-Abs test set and a large majority of documents in the UK-Abs test set are actually shorter than 8,192 words; hence the summary is obtained at once by passing the entire document through the chatgpt-16k model.

Let $\lvert Doc\rvert$ be the length of an input document, and let $\lvert Exp\rvert$ be the length of its reference summary written by human experts. Assume that the document is split into a series of chunks, each of size $K$ words. Then the target summary length to be generated by the model for each model is $K\times\lvert Exp\rvert/\lvert Doc\rvert$ words. In other words, we use the LLMs / summarization models to summarize every chunk considering the same compression ratio as for the whole document and the reference summary.

There is an inherent limitation in this simple method. In reality, all parts of the document are not equally important, hence different chunks should ideally be allocated different lengths in the summary. In contrast, this method allocates the same length in the summary for all chunks. However, there is no simple way of knowing the relative importance of different chunks in a legal case judgement; hence we follow the simple method of assuming the same compression ratio for every chunk.

Here we measure the match of an algorithm-generated summary for a document with the gold standard / reference summary of the same document. The following metrics are used to measure the performance of the summarization models.

Now we discuss three metrics used to assess the consistency of model-generated summaries. These metrics compare a model-generated summary with the original document and estimate how consistent the summary is with the document. All these metrics give a score in the range [0, 1]; the higher the score, the more consistent is the summary.

Given a (document, summary) pair, the SummaC metric [15] computes NLI scores for each sentence in the model-generated summary, indicating the likelihood that the sentence logically follows from the sentences in the original document. Lower NLI scores for a particular sentence $s$ in the summary suggest a higher mismatch between this sentence and the document, and hence the potential presence of hallucinated information. The NLI scores of individual sentences in the summary are combined to give a single SummaC score for the given (document, summary) pair. A higher SummaC score indicates greater consistency between the model-generated summary and the original document. We use the standard implementation of SummaC available at https://github.com/tingofurro/summac.

First we compare the performances of Chatgpt, Davinci, and Llama with different chunk sizes, to check which chunk size gives the best performance. We perform these experiments on the IN-Abs dataset.

Table 3 shows the ROUGE, METEOR, BERTScore metrics for chatgpt-summ, chatgpt-16k-long, davinci-summ, and llama-summ, using different chunk sizes. The same prompt is used for all the different variations of Chatgpt, Davinci, and Llama, as stated in Section 4.2. The highest value for every metric is shown in blue-bold.

For all the three LLMs (chatgpt, davinci and Llama), the best results for most metrics are obtained with chunk-size 1024. This is possibly because the LLMs find it difficult to capture the context when the chunk sizes become too large, thus leading to better summaries with chunk size 1024. However, we preferred not to use even shorter chunks, since there is a possibility of redundancy and loss of continuity/coherence if the document is broken into too many chunks. Hence, we perform all subsequent experiments with chatgpt, davinci, and Llama considering a chunk size of 1024 words.

For chatgpt-16k, we will continue to use longer chunks of 8192 words. For GPT-4 Turbo, we feed the entire document into the model since GPT-4 Turbo has a very long context length of 128K.

We first compare the performances of LLM-based summarization using different prompts (that were stated in Section 4.2), and select the best LLM variants for summarization of this dataset. Then we compare the best LLM variants with other extractive and abstractive summarization models.

We observe that the ‘hybrid’ variants have higher Rouge-2 precision and Rouge-L precision across all the model variants. This suggests that the summaries generated by the ‘hybrid’ variants contains many of the same sequences of words (in the same order) as the reference / gold standard summaries. Whereas, the ‘explicit’ and ‘summ’ variants have higher Rouge-2 recall and Rouge-L recall across all the model variants, suggesting that the summaries include a significant portion of the content of the reference summaries.

An interesting observation is that the summaries created by GPT-4 Turbo are generally shorter in length, compared to the summaries created by Turbo-GPT-3.5 (Chatgpt). This is why the summaries created by Chatgpt have higher matches with the gold standard summaries, thereby having higher scores for the Recall and FI metrics. However, the summaries generated by GPT-4 achieve higher Rouge-Precision scores in most cases.

We now compare some of the best performing LLM variants with the other types of summarization models.

Amongst the extractive summarization models, CaseSummarizer has performed better than other extractive methods across most metrics. Among the legal domain-specific abstractive models, LegLED achieves slightly higher ROUGE-2 and ROUGE-L Recall and F1 scores, while LegPegasus achieves higher METEOR and BERTScore values, and ROUGE-2 and ROUGE-L Precision scores.

Across all methods, the highest Rouge-2 recall, Rouge-2 F1 score, Rouge-L F1 score, and the highest METEOR score are achieved by davinci-summ. We see that the general-domain LLMs (davinci-summ in particular) perform better than the extractive models for the UK-Abs dataset.

The highest Rouge-2 Precision and Rouge-L Precision are obtained for LegPegasus, whereas LegLED achieves higher Rouge-2 Recall and Rouge-L Recall scores. We observe that the summaries generated by LegPegasus are generally shorter than the summaries generated by LegLED for the UK-Abs dataset. Specifically, the average length of summaries generated by LegPegasus is 820.77 words, while that of LegLED is 890.63 words. This difference in length of the summaries plays a role in higher ROUGE precision scores for LegPegasus and higher ROUGE Recall scores for LegLED.

Among the chatgpt variants, chatgpt-tldr gets the highest NEPrec and NumPrec scores, while chatgpt-hybrid gets the highest SummaC score. Among the gpt4 variants, gpt4-tldr gets the highest NEPrec scores, while gpt4-hybrid gets the highest SummaC and NumPrec scores. Among the llama variants, llama-tldr gets the highest NEPrec scores, while llama-hybrid gets the highest NumPrec score and llama-explicit gets the highest SummaC score. Whereas, among the davinci variants, davinci-summ gets the highest NEPrec and NumPrec scores, and the highest SummaC score is obtained by Davinci-explicit (which is the overall highest score for SummaC). The overall highest values for NEPrec and Numprec scores are achieved by GPT4-Turbo (NEPrec jointly highest with chatgpt-16k-long).

From Table 6, it is evident that there are instances of hallucination / inconsistency in the summaries generated by LLMs and other generative summarization models. We will see some actual examples of such inconsistencies in the next section.

We now study the performances of various summarization models on the IN-Abs dataset. As we did in the previous section for the UK-Abs dataset, here also we first compare the performances of LLM-based summarization using different prompts (that were stated in Section 4.2), and select the best LLM variants for summarization of this dataset. Then we compare the best LLM variants with other extractive and abstractive summarization models.

Amongst the chatgpt models, chatgpt-summ achieves the best value for most metrics, including METEOR and BERTScore, although chatgpt-explicit achieves the best ROUGE-L Recall and F1 scores. Amongst the gpt4 models, gpt4-summ achieves the best value for most metrics, including METEOR and BERTScore, although gpt4-explicit achieves the best ROUGE-L Recall and F1 scores. Among the llama models, different variants get the highest score for different metrics. Among the davinci models, davinci-summ achieves the highest values for most metrics.

We observe that the ‘tldr’ variants have higher Rouge-2 precision and Rouge-L precision across all the model variants. Whereas, the ‘summ’ and ‘explicit’ variants have higher Rouge-2 and Rouge-L recall and F1 scores across all the model variants.

An interesting observation is that the summaries produced by GPT-4 Turbo are shorter than those generated by Turbo-GPT-3.5 (Chatgpt). As a result, the summaries from Chatgpt exhibit higher ROUGE Recall and F1 scores, whereas the GPT-4 summaries achieve higher ROUGE precision scores.

We now compare some of the best performing LLM variants with the other types of summarization models.

Among the extractive models, CaseSummarizer achieves the highest metric values for most metrics, especially the ROUGE-based metrics. In fact, for most ROUGE-based metrics, CaseSummarizer achieves the overall highest values for all metrics. Among the legal domain-specific abstractive models, LegPegasus performs better than LegLED across all metrics. Among the LLMs, chatgpt-summ achieves higher ROUGE-L Recall and BertScore values than the extractive models. Thus for the IN-Abs dataset, it is seen that the abstractive models (including LLMs) perform at par with the best extractive models.

Among the chatgpt variants, chatgpt-hybrid gets the highest SummaC and NEPrec scores, while chatgpt-explicit gets the highest NumPrec score. Among the gpt4 variants, gpt4-hybrid gets the highest SummaC scores, while gpt4-explicit gets the highest NumPrec score (which is the overall highest score for NumPrec) and gpt4-summ gets the highest NEPrec scores. Among the llama2-70b variants, llama-hybrid gets the highest SummaC and NEPrec scores, while llama-explicit gets the highest NumPrec score. Among the davinci variants, davinci-explicit gets the highest SummaC (highest overall) and NumPrec scores, and the highest NEPrec score is obtained by davinci-hybrid.

Again, it is evident that there are instances of hallucination / inconsistency in the summaries generated by LLMs and other generative summarization models. We will see some actual examples of such inconsistencies in the next section.

As stated earlier, the abstractive summarization models LegPegasus and LegLED are originally fine-tuned over legal (case judgement, summary) pairs from courts in the USA. These models allow further fine-tuning. Hence we check if domain-specific fine-tuning (i.e., fine-tuning over data from the target domain) of these models helps in improving consistency of the generated summaries.

To make these models more suitable for summarizing UK legal documents, we further fine-tune them on the UK-Abs training dataset containing 693 UK case judgements and their corresponding reference summaries. These models fine-tuned on the UK-Abs training dataset are referred to as LegPegasus-UK and LegLED-UK. Similary, to make these models more suitable for Indian case judgements, LegLED and LegPegasus models are further finetuned on the IN-Abs training dataset consisting of 7030 (case judgement, summary) pairs. We refer to these models fine-tuned over Indian data as LegLED-IN and LegPegasus-IN.

Table 14 shows the ROUGE, METEOR, and BERTScore for legal domain-specific abstractive models on the IN-Abs dataset before (LegPegasus and LegLED) and after (LegPegasus-IN and LegLED-IN) using fine-tuning. Table 15 shows the consistency metrics over IN-Abs for the same scenarios. It is seen that both the match of the generated summaries with reference summaries (Table 14) as well as the consistency of generated summaries (Table 15) improve after fine-tuning over data from the target domain.

Similarly, Table 16 and Table 17 compare the performance of the abstractive summarization models on the UK-Abs dataset, before and after fine-tuning over target domain data. We again see that performance as well as consistency of the generated summaries improves massively upon fine-tuning over the target domain data.

Recall that when LegLED was used to summarize Indian or UK judgements, there were several instances of US court and statute names being hallucinated in the summaries (e.g., the examples stated in Table 12 and Table 13). Importantly, we did not observe such instances in the summaries generated by LegLED-IN or LegLED-UK. While the summaries generated by LegLED-IN or LegLED-UK also contain some issues such as words or sentences being merged together, we did not observe such gross hallucinations in the summaries generated by these fine-tuned versions.

Thus, for both datasets IN-Abs and UK-Abs, it is seen that fine-tuning over the target domain data helps to improve the quality as well as consistency of the abstractive summaries. If data in the target domain is available, it seems an effective approach to utilize such data for fine-tuning abstractive summarizers.

Unlike the abstractive summarization models described previously, LLMs are very difficult/expensive to fine-tune. Hence, avoiding hallucinations in LLMs is much more challenging and is, in fact, an open question [48].

In this work, we explore a very simple approach to reducing hallucinations in LLMs – we include in the prompt an instruction to avoid hallucinations / inconsistencies. Also, since we observed the presence of incomplete sentences in the generated summaries, we included an instruction to output complete sentences only. Specifically, we add the following text in the prompt – “Output complete sentences and not half sentences. Do not have hallucinations and inconsistencies in your summary.”

We selected those variations of the LLMs that performed the best on the two datasets (e.g., chatgpt-explicit, davinci-summ, and chatgpt-16k-long for the UK-Abs dataset), and created new variations by adding the above-mentioned text in the prompt. For instance, we got a new variation from chatgpt-explicit, which we call Chatgpt-reduce-hallucination (abbreviated as chatgpt-RH, by using the prompt: “Your task is to summarize the following document in at most <YY> words. Output complete sentences and not half sentences. Do not have hallucinations and inconsistencies in your summary. The document to be summarized is given within <>. Document to summarize - <text to summarize>”. Similarly, we got the variations Davinci-reduce-hallucination (abbreviated as davinci-RH) from davinci-summ, and chatgpt-16k-RH-hallucination (abbreviated as chatgpt-16k-RH) from chatgpt-16k-long, by including the explicit instructions for reducing hallucinations and outputing complete sentences.

For the IN-Abs dataset, Table 18 shows the ROUGE, METEOR, and BERTScore metrics for the different variations of the LLMs while Table 19 shows the consistency metrics for the different variations. Similarly, for the UK-Abs dataset, Table 20 shows the ROUGE, METEOR, and BERTScore metrics while Table 21 shows the consistency metrics for the different versions of the LLMs. Across both datasets, we find that the RH versions score lower than the ‘explicit’ or ‘summ’ versions of the LLMs, according to most ROUGE-based, METEOR and BertScore metrics. Whereas, the RH versions score higher according to the consistency metrics (SummaC, NEPrec, NumPrec).

We manually examined several summaries generated by the RH variants and the corresponding summaries by the ‘summ’, ‘long’ and ‘explicit’ variants, to understand the reasons behind the fall in the summary quality of the RH variants. We observed that the summaries generated by the RH-variants have lesser amount of key information, such as named entities (names of persons, organizations, etc.) in the case document, as compared to those generated by the other variants. For instance, the average number of named entities in the summaries generated by chatgpt-RH is 13.9, while that in the summaries generated by chatgpt-summ is 16.5. The summaries generated by the RH-variant have lesser amounts of unigram, bigram, longest common subsequence, and semantic similarity matches with the gold standard summaries as compared to the ‘summ’ variant. Since there is lesser amount of key information contained in the summaries generated by the RH variants, the ROUGE, METEOR, and BERTScore of these summaries are lower than those for the other variants.

These observations imply that, while it is possible to reduce hallucinations / inconsistencies in the generated summaries by explicit prompting, there may be an associated reduction in the quality of the summary. A future direction of research would be to devise strategies for reducing hallucinations and inconsistencies in LLM-generated legal summaries, while maintaining the information quality of the summaries.

We now describe a novel approach that uses semantic similarity between entities to reduce hallucinations and inconsistencies in summaries. Given a legal judgement $j$ and its summary $s$ generated by an AI model, the approach consists of the following steps:- (i) Consider all the named entities and numbers present in the legal judgement $j$. Let the set of named entities and numbers present in $j$ be $V_{j}$ (the set of named entities and numbers can be detected by any standard named entity recognizer; in particular, we use the popular Python Spacy library). (ii) Consider all the named entities and numbers present in the summary $s$ generated by the AI model. Let the set of named entities and numbers present in the summary be defined as $V_{s}$. (iii) Now define a set $V_{r}=V_{s}-V_{j}$. $V_{r}$ is the set of all those named entities / numbers that are present in the generated summary but are not present in the original judgement (hallucinated entities). (iv) For every element in the set $V_{r}$, consider an embedding/representation from any pre-trained language model (specifically, we used the bert-base-uncased embeddings). Also, for every element in the set $V_{j}$, consider the same embedding. (v) Calculate the cosine similarity between the embeddings of every element in $V_{r}$ and every element in $V_{j}$. (vi) Replace every element in the set $V_{r}$ with that element in $V_{j}$ which has the highest cosine similarity with this element, based on the embeddings (numbers to be replaced by numbers only). Basically, we replace every hallucinated named entity or number (in the generated summary) with the most semantically similar entity that is present in the input document.

The chatgpt variant which uses this semantic similarity based approach to reduce hallucinations is denoted as chatgpt-SS. Similarly, we denote the Davinci and chatgpt-16k variant which uses the semantic similarity based approach to reduce hallucinations as davinci-SS and chatgpt-16k-SS respectively.

Table 22 shows examples of two summary extracts with hallucinated named entities / numbers replaced by the correct entities using the semantic similarity based method. The first example shows the wrong monetary amount ‘Rs. 26,500’ replaced by ‘Rs. 29,500’. The second example shows that a wrong judge name is replaced by the correct judge name using the semantic similarity based method.

These observations imply that, there is a reduction in hallucinations / inconsistencies in the generated summaries by this approach, and there is also an improvement in terms of results of the automated metrics. Hence, the semantic similarity-based approach is quite effective in improving the quality of generated summaries. However, we have noticed that all hallucinations / inconsistencies cannot be corrected by this approach; hence the correction of hallucination/inconsistency in abstractive summaries remains an open area of research.

The GOVREPORT dataset, obtained from [49], comprises of 19,466 detailed reports from the U.S. Government Accountability Office (GAO) and the Congressional Research Service (CRS). These reports, prepared in response to congressional requests, span a broad range of national policy topics and include human-written summaries. During data collection, some boiler-plate parts were removed from the crawled files, and the section and paragraph structure of the documents and summaries were retained. The dataset features 12,228 GAO reports and 7,238 CRS reports. On average, GAO reports have 6.9 sections, while CRS reports have 4.6 sections. The dataset is organized into training, validation, and test sets based on the publication dates, resulting in 17,519 training samples, 974 validation samples, and 973 test samples. Table 27 shows the dataset statistics for the GOVREPORT dataset. Following [42, 43], the coverage and density of the legal judgements with respect to the gold-standard summaries is 0.16 and 1.40 respectively for this dataset.

In this work, the summarization models are evaluated over a randomly sampled 100 document-summary pairs from the test set of GOVREPORT. The supervised summarization models are trained over the training set of GOVREPORT.

We now study the performances of summarization models on the GOVREPORT dataset. To this end, we report the results of some of the best performing LLMs and summarization models (as observed in the previous sections of this paper). From the GPT-4 Turbo variants, we report the results of the best performing variant gpt4-summ. Similarly, from the Llama2-70b and Chatgpt variants, we report the results of llama-summ and chatgpt-summ. We also report the results of LegPegasus and LegLED fine-tuned on the training dataset of GOVREPORT; we denote these models as LegPegasus-GR and LegLED-GR. We also report the results of best performing extractive summarization model namely CaseSummarizer.

In general, we see higher metric values, especially for the ROUGE scores, for GOVREPORT than what we obtained for the IN-Abs and UK-Abs datasets. These trends seem to indicate that summarizing legal case judgement reports is a more complicated task than summarizing the Government reports in the GOVREPORT dataset.