Rapid Biomedical Research Classification:
The Pandemic PACT Advanced Categorisation Engine

The surveillance and monitoring of emerging and re-emerging pathogens are vital to global health security. Infectious diseases with the potential to cause pandemics represent a significant threat to public health, economies, and societies worldwide. Efficiently tracking these threats and coordinating research efforts is essential to mitigate their impact. Traditional approaches to research funding and coordination during health crises, such as the COVID-19 pandemic, often exhibit limitations including slow activation of research projects, duplication of efforts, and fragmented funding landscapes. These issues highlight the need for improved systems to manage and analyse research activities (Carroll et al., 2021; McLean et al., 2022).

The rapid identification and response to infectious disease threats require a well-coordinated and systematic approach to research funding and project tracking. Accurate categorisation and analysis of research projects are crucial for identifying trends, understanding research gaps, and ensuring that resources are allocated effectively. This task is inherently complex, given the diverse and interdisciplinary nature of biomedical research (Seminog et al., 2024).

Artificial intelligence (AI), particularly large language models (LLMs), offers a promising solution to enhance the efficiency and accuracy of research categorisation. LLMs can be fine-tuned to assist in the classification of research abstracts, providing valuable support to human annotators. These models not only streamline the categorisation process but also generate rationales for their decisions, adding a layer of interpretability and transparency that is essential for gaining the trust of researchers and policymakers³³3LLMs are typically seen as complex and opaque, and their interpretability is a nuanced, ongoing research topic (Luo and Specia, 2024). Here, we use “interpretable” to mean that medical practitioners find the process more engaging and understandable than with encoder-only models, as LLMs can explain their reasoning in human language..

This paper introduces the Pandemic PACT Advanced Categorisation Engine (PPACE), a fine-tuned LLM designed to classify biomedical research abstracts according to WHO-aligned research priorities. PPACE leverages human-annotated data and employs generative AI to produce rationales for each classification. By automating the categorisation process, PPACE aims to enhance the monitoring of research trends and the identification of critical gaps in global health preparedness and response.

In the remainder of the paper, we first provide an overview of the literature on the use of LLMs in biomedical document classification (Section 2). Next, we introduce the Pandemic PACT project which this work builds on, and describe the details of the dataset and the annotation procedure involved in its creation (Section 3). Section 4 will describe the methodology in finetuning the PPACE model, and finally, in Section 5, we present the results and conclude the paper. The contributions of this work are as follows:

1. 1.

   We contribute to the task of biomedical document classification by publicly releasing a carefully annotated dataset of research projects (each project containing a title and a PubMed-style abstract) gathered as part of the Pandemic PACT project and further preprocessed to include rationales generated by a 70B LLM. The augmented dataset is formatted as an instruction-based dataset and can be used to train similar models by the research community.

2. 2.

   We fine-tune and publicly release an 8B model trained on the aforementioned dataset and make the model weights available publicly.

3. 3.

   We perform a range of analyses on the dataset to shed light on the complexities of the data and run a number of evaluations to ensure that the model outperforms the baseline.

Biomedical document classification is an active area of research that has attracted considerable attention in recent years (Laza et al., 2011). The PubMed 200k RCT dataset, for instance, focuses on classifying different sections of randomized controlled trial abstracts into categories like objectives, methods, results, and conclusions (Dernoncourt and Lee, 2017). Another notable task in this area is the Hallmarks of Cancer (HoC), which presents a multi-label classification challenge and aims to identify key cancer-related research themes from PubMed abstracts (Baker et al., 2015). The LitCovid dataset (Chen et al., 2020; Jimenez Gutierrez et al., 2020), which comprises over 30,000 COVID-19-related articles, each annotated with one or more topics relevant to the pandemic, is another major biomedical document classification benchmark that has been studied in the literature. Automated topic annotation tasks like this can significantly reduce the manual curation burden during pandemics, and the present work can be considered a more generalized effort to automate the classification of biomedical literature into research themes of interest. Additional datasets relevant to biomedical document classification include the BioCreative Corpus III (BC3, Arighi et al., 2011) and TREC (Hersh et al., 2006). Behera et al. (2019) provides an overview of this task and the various deep learning algorithms to address it.

Large Language Models (LLMs) have become ubiquitous in various text processing and classification tasks, including document classification. Their ability to handle a wide range of text-related tasks makes them particularly appealing for numerous applications. LLMs can be instructed to perform specific tasks via few-shot examples or through fine-tuning with detailed instructions. For biomedical researchers, generative LLMs are especially valuable because they can be interfaced with using natural human language, facilitating more intuitive and effective interactions.

SciFive (Phan et al., 2021) is a domain-specific T5 model that has been pretrained to address a number of biomedical tasks, including document classification. Rohanian et al. (2023) is the first attempt to use generative language models to address classical biomedical text processing tasks like HoC via instruction tuning. Chen et al. (2023) and Tian et al. (2024) have studied the use of LLMs in a number of biomedical text processing tasks, including document classification, although the focus is mostly on closed-source frozen models like GPT-4.

Various techniques are observed in the literature regarding the use of LLMs when addressing this task. Several studies use parameter-efficient fine-tuning methods (Hu et al., 2021; Taylor et al., 2024; Jiang et al., 2024) which have become very prevalent due to the ease of use and faster training time they offer. Our work not only employs instruction tuning and LoRA as a parameter-efficient fine-tuning technique, but also draws inspiration from Hsieh et al. (2023) in that it utilises ‘rationales’ generated by a larger model to augment the labelled dataset and then fine-tunes a smaller one trained on the expanded dataset.

In this work, we initially use a manually labelled dataset to generate ‘rationales’ for the labels using a Llama-3 70B model⁶⁶6https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct. These rationales explain why each category label is chosen, pinpointing the reason by referencing the abstract. We add these rationales to the labels, constructing an expanded dataset that includes both the prompt and the rationales. For details on the prompt template used, refer to Appendix Section B.

We subsequently explore adapting a smaller model to the classification task using this dataset. The fine-tuned model is expected not only to predict the labels but also to explain its reasoning. Based on insights from McCoy et al. (2023) regarding the limitations of autoregressive language models, we structured the prompt such that the rationales are provided first before the model determines the category labels. This approach ensures that errors from irrelevant categories are not propagated back into the model’s outputs. Given the substantial computing resources required to train the full weights of the 8B model, we explored the use of efficient fine-tuning via LoRA (Section 4.2).

We have chosen to restrict the experiments and the baselines to a single decoder-only transformer. As of this writing, an LLM with around 10 billion parameters is considered relatively small but can be performant enough to rival state-of-the-art. A representative LLM with a good starting performance on this complex task provides us with a foundation to improve upon while avoiding potential saturation. Additionally, a very large model like Llama-3 70B would be impractical for independent researchers to fine-tune or utilise on less powerful machines. Section 4.1 details how this LLM was chosen.

Table 6 shows the results of the baseline Llama3-8B model, the larger Llama3-70b model, and the proposed PPACE model, respectively. As can be seen, PPACE outperforms the baselines on all the metrics with the exception of macro-averaged recall.

The finetuned model demonstrates significant improvements in F1 scores across most categories, indicating the effectiveness of the finetuning process. Notable improvements are seen in categories like ‘Infection Prevention & Control’, ‘Therapeutics’, and ‘Vaccines’, where the finetuned model’s precision and F1 scores show substantial gains. Categories with low representation in the dataset, such as ‘Capacity Strengthening’ and ‘Health Systems’, see mixed results, with some performance metrics slightly decreasing⁹⁹9The distribution of each category in the test set is provided in the Appendix Section C. This suggests that while finetuning enhances the model’s ability to generalise, it may still struggle with categories that have very few examples. Figure 4 shows the changes in F-scores between the fine-tuned and the base model across the different categories, sorted from the least frequent to the most frequent as seen in the test set.

Overall, the finetuned model generally performs better in terms of precision compared to recall. This trend indicates that the model has become more conservative in its predictions, leading to fewer false positives but potentially more false negatives. Categories like ‘Vaccines’, ‘Public Health’ and ‘Secondary Impacts’ show remarkable improvements in precision and F1 scores, demonstrating PPACE’s enhanced capability to identify relevant instances within these categories. The dramatic increase in all metrics for ‘Public Health’ is particularly noteworthy, with the F-score jumping from 0 to 0.65. However, there remains room for improvement, especially for categories with minimal representation in the dataset. The results highlight the strengths of the finetuning approach while also pointing out the difficulty of the task.

In this work we introduced the Pandemic PACT Advanced Categorisation Engine model or PPACE, a fine-tuned 8B language model for biomedical research classification as part of the Pandemic PACT initiative. PPACE is capable of accurately categorising research abstracts according to WHO-aligned priorities. This tool can be a valuable asset for identifying biomedical research trends and gaps in a multilabel classification scenario. The model was built on a robust foundation of human-annotated data, enhanced with LLM-generated rationales, ensuring that the model’s predictions are not only accurate but also interpretable. The use of efficient fine-tuning has enabled us to adapt the model effectively while maintaining computational efficiency.

Our evaluation demonstrated that PPACE outperforms its baselines, offering significant improvements in the context of multilabel classification. The model and the instruction-based dataset used for training are released oublicly, providing a valuable resource for the research community. These contributions facilitate further advancements in aligning biomedical research with critical global health priorities.

Looking ahead, the integration of LLMs in the annotation process promises to streamline data collection and categorisation, potentially reducing the burden on human annotators and improving the scalability of such initiatives. The evolution of PPACE can play a crucial role in enhancing the efficiency and effectiveness of global health research, ultimately contributing to better preparedness for future outbreaks.

Our work has several limitations. First, the dataset used for training and evaluation, while extensive, may not encompass the full diversity of biomedical research projects globally, potentially limiting the generalisability of our model for prospective analyses of research in new emerging pathogens. Additionally, the research categories might become outdated at some point, requiring updates and subsequent retraining of the model. Second, some projects can be categorised in different ways, introducing a degree of subjectivity in certain assignments. While our use of human-annotated expert labels aims to minimise this issue, it does not completely eliminate it.

Furthermore, despite using efficient fine-tuning methods like LoRA, the 8-billion parameter model is still sizable. Researchers with limited computational resources would need reliable GPUs for inference, as running solely on CPU can be very slow. Future iterations of this work will aim to fine-tune smaller models to improve accessibility.

Lastly, we did not experiment heavily with advanced prompting techniques or invest significant time in crafting the best possible prompts. There is potential for further improvement in the reported results for the frozen language models through optimised prompts, which might narrow the performance gap with the fine-tuned model.

This work was primarily funded by Wellcome [226543] and the EDCTP2 Programme supported by the European Union. The Pandemic PACT Programme is also supported by the following grants: This research was funded by the National Institute for Health Research (NIHR) (CSA2022GloPID-R-3387) using UK Aid from the UK Government to support global health research. This work was carried out with the aid of a grant from the International Development Research Centre, Ottawa, Canada (109910-001). This work was supported by UK Research & Innovation (UKRI) under the UK Government’s Horizon Europe Guarantee under GloPID-R SEC 3 Grant Agreement no. 10061268. Whilst the funders of Pandemic PACT are engaged through the Pandemic PACT Advisory Group and have a role in the provision of funding data, they are not involved in the analysis and presentation of related findings.

This work was supported in part by the National Institute for Health Research (NIHR) Oxford Biomedical Research Centre (BRC), and in part by an InnoHK Project at the Hong Kong Centre for Cerebro-cardiovascular Health Engineering (COCHE). OR acknowledges the support of the Medical Research Council (grant number MR/W01761X/). DAC was supported by an NIHR Research Professorship, an RAEng Research Chair, COCHE, the UKRI, and the Pandemic Sciences Institute at the University of Oxford. The views expressed are those of the authors and not necessarily those of the NIHR, MRC, COCHE, UKRI, or the University of Oxford.