Enabling Discriminative Reasoning in LLMs for Legal Judgment Prediction

We first formally describe the workflow of legal judgment prediction. Given the criminal fact $f$ from a real case, the name of one of the defendants, a set of charges $\mathcal{C}$, and a set of law articles $\mathcal{A}$, our task is to predict the applicable subset of charges $\mathcal{C}_{d}$, the relevant subset of law articles $\mathcal{A}_{d}$, and the term of imprisonment for the defendant $d$. To align with legal practice, we follow recent studies that treat charge and law article prediction as multi-label classification tasks, and term of imprisonment prediction as a multi-class classification task.

We propose a discriminative reasoning framework, called ADAPT, to guide LLM to gradually deduce the most appropriate charges and law articles step by step, including Ask, Discriminate, and Predict. As shown in Figure 2, in the first step, Ask, the model is prompted to identify the key legal elements that constitute crimes through a question-answering approach. Then, in the second step, Discriminate, we prompt LLM to utilize the extracted key elements from the Ask step to initially identify the top-$K$ most possible charges, and subsequently evaluate the consistency between each charge and the established facts. Finally, in the Predict step, LLM integrates the reasoning signals from the previous two steps to identify the most suitable charges and law articles. We describe the details of these three steps in the following.

While our ADAPT prompting framework outperforms traditional prompting methods such as direct reasoning or legal syllogism, we find that current general LLMs still struggle to consistently generate fully accurate reasoning trajectories. This limitation arises from their restricted legal-specific knowledge and lack of exposure to our specialized ADAPT reasoning patterns. Additionally, we find that current LLMs often reject giving the sentencing ranges, likely because of their strict safety alignment. To address these issues, we propose fine-tuning a better LLM under our ADAPT framework for more comprehensive, efficient, and effective legal judgment prediction.

We conduct extensive experiments in both single-defendant and multi-defendant scenarios to comprehensively evaluate our method’s performance in real-world applications. In the single-defendant context, we employ the widely-used CAIL2018 dataset Xiao et al. (2018). For the multi-defendant case, we select the MultiLJP Lyu et al. (2023) dataset whose labels are verified by human experts. Both datasets are divided into 11 intervals to convert prison term prediction to a multi-class classification task. Detailed statistics of both datasets are provided in Table 1. For evaluation metrics, we follow previous works to adopt Accuracy (Acc.), Macro Precision (Ma-P), Macro Recall (Ma-R), and Macro F1 (Ma-F) across all sub-tasks.

For all LLM-based fine-tuning methods, we utilize Qwen2-7B as the foundation model. LoRA is adopted for parameter-efficient fine-tuning of the large language model. We apply LoRA to all linear modules of the model, with both alpha and rank set to 32. The language modeling head is also unfrozen to enhance learning. Our model is fine-tuned by 10 epochs, with a learning rate of 5e-5 and a batch size of 64. The total number of training reasoning trajectories for CAIL2018 and MultiLJP is 80,141 and 157,763, respectively. Greedy decoding is used for all generative models to enhance the stability of results. For those generated charges that are not present in the label pool, we use BGE Xiao et al. (2023) to map them to the closest charge in the pool based on their representations.

The results on the fine-tuning setting are presented in Table 2. We have the following findings:

(1) Our method outperforms all baselines across all metrics on both datasets. Overall, the LLM-based approaches show superior performance, indicating that large causal language models can adapt effectively to the tasks of legal judgment prediction with targeted fine-tuning. On the CAIL2018 dataset, our approach achieves relative accuracy and Ma-F improvements of 4.1% and 4.7%, respectively, over Finetune-CoT. This demonstrates that our synthetic data significantly enhances the effectiveness of legal judgment prediction.

(2) The ADAPT framework achieves a more significant advantage in charge and law article prediction. These two tasks are both multi-label classification tasks, meaning a successful shot satisfies the predicted set to be completely consistent with the label set. Our improvements suggest that discriminative reasoning offers more evidence for the precise inference of the target set. And on the other side, the performance across methods is relatively close in prison term prediction. We hypothesize that this is due to the uncertainty introduced by real-world judges’ discretion, which cannot be fully mitigated by classification metrics based on rigid interval divisions.

(3) Fine-tuning with discriminative reasoning trajectories further enhances the language models’ performance. Finetune-CoT similarly utilizes synthetic reasoning data to promote autoregressive learning. However, the synthesized CoT data is observed to result in sub-optimal performance. We argue that this is because the “step-by-step reasoning” generated by the language model does not contribute to more supporting logic. Instead, it merely discusses the plain facts in most cases.

We evaluate under the prompting setting using the CAIL2018 dataset, as the MultiLJP dataset contains only 23 charges and can not reflect the effect of confusing charges in real-world scenarios. During our experiments, we observed that the in-domain model fails to effectively adhere to instructions for few-shot learning, so we only tested its zero-shot ability. Moreover, we discard the prison term prediction task because LMs typically refuse to predict the accurate terms. The results are reported in Table 3, from which we can observe:

(1) Our approach generally demonstrates superior performance across models of different sizes, suggesting that prompts can still activate discriminative reasoning to some extent. Moreover, we also observe that the improvements are not pronounced as in the fine-tuning setting. This indicates that fine-tuning can further enhance the model’s capabilities within our framework.

(2) Larger models typically bring better performance, but they still exhibit a notable disparity when compared to models trained for LJP specifically. This might be because recent leading open-source models have been trained on extensive data from major domains. This suggests that fine-tuning remains a valuable strategy for LLMs to adapt to the requirements of LJP tasks

We investigate the effects of various reasoning data on the final LJP tasks within the fine-tuning context. The specific ablation strategies are described as follows: (1) w/o Ask: The task of summarizing legal elements from facts is removed, and $y^{\text{Ask}}$ is excluded from $y^{\text{All}}$. (2) w/o Disc: After the Ask step, the language model must directly predict the charges and law articles. (3) w/o Article: We remove the law article-related reasoning trajectories in the training data. (4) w/o Sentencing: The language model no longer analyzes sentencing factors before determining the prison term prediction. (5) ADAPT$\rightarrow$Refine: We construct candidate items for charges and law articles and provide them to the large language model, requiring it to refine them and determine the final prediction. During inference, the candidates are derived from the top-k items in the probability distribution of BERT, while during training, the correct labels are ensured to be included among these candidates.

The ablation results are shown in Table 4. Firstly, we can observe the removal of each sub-task leads to a performance decline, indicating that each type of synthetic data makes a positive contribution. Additionally, we notice that the ablation associated with law articles causes obvious impact. This impact may be attributed to the better alignment between legal provisions and facts of real cases, which enhances legal reasoning abilities in other forms. Finally, it is observed that the ablation related to sentencing almost exclusively affects prison term prediction. This can likely be attributed to the fact that sentencing factors and conviction factors are orthogonal in most cases.

Moreover, we find that ADAPT$\rightarrow$Refine results in a significant performance decline. This is because the top-k candidates provided by the domain model (i.e., BERT) during inference often do not include the ground truth labels, thus the LLM can only select relatively close labels in such cases. We think that the fine-tuned LLMs are inherently capable of generating high-quality candidates and don’t require assistance from smaller external models.

Instruction tuning has been observed to lead to performance degradation with an increasing number of training epochs. However, previous studies demonstrate that it often requires many epochs (e.g., 20 epochs) for model performance to coverage in LJP tasks. We investigate the effects of training epochs for the ADAPT framework in this section. The results, presented in Figure 3, reveal that:

(1) In the predictions of charges and law articles, metrics gradually increase with the number of training epochs and stabilize after reaching a peak at a certain epoch. This indicates that the model can continuously learn effective features from the data even after the initial complete iteration.

(2) The metric curves for different sub-tasks exhibit notable distinctions. Unlike the other two tasks, most of the metrics for prison term prediction show a marked decline after the third epoch. This divergence highlights the inherent difficulty in predicting prison terms. Judicial discretion in sentencing can span multiple pre-defined intervals, thereby increasing the risk of the model overfitting when learning from repeated samples.

(3) In multi-label classification tasks (i.e., charge and law article prediction), the model consistently exhibits higher performance in the Macro-Recall compared to the other three metrics. This indicates that the fine-tuned language model tends to identify more possible positive candidates. We believe this also suggests the potential for further refining results in our proposed reasoning framework.

Exploring performance across different charges can help us better understand the detailed improvements achieved by our method. We first calculate the F1 scores of the finetuned BERT model for each charge in the CAIL2018 dataset and rank these charges from highest to lowest. For clearer visualization, we categorize the ranked charges into four sets and then evaluate the macro-F1 scores of all finetuned models on the charge prediction task. Generally, the charges in the higher quartiles (e.g., 75%-100%) exhibit greater prediction difficulty, as evidenced by the poor performance of mainstream "Text-to-Label" style models on these charges. The experimental results are shown in Figure 4, from which we can observe the following findings:

(1) Our ADAPT framework achieves a more significant improvement on difficult sets. Specifically, in the 75%-100% interval, ADAPT achieves relative improvements of 15.1% and 16.7% compared to mT5 and Vanilla-SFT, respectively. Conversely, in the 0%-25% interval, the relative improvements are merely 6.5% and 1.4%. This suggests that our discriminative reasoning approach effectively delineates subtle differences between various charges, thereby significantly enhancing prediction accuracy for more confusing charges.

(2) The marginal benefits of employing simple instruction fine-tuning on larger models are limited. We can observe that despite Vanilla-SFT leveraging a language model with 7B parameters, its improvements over mT5 are not substantial. Notably, in cases involving more difficult charges, Vanilla-SFT is even likely to demonstrate a slight decline in performance. This finding highlights the importance of identifying an appropriate reasoning pattern to enhance the effectiveness of large language models in legal judgment prediction.