Panacea: A foundation model for clinical trial search, summarization, design, and recruitment

Our goal is to develop Panacea, a domain-specific foundation model for clinical trial tasks. Like previous works on developing domain-specific foundation models [23, 24], the biggest challenge for developing Panacea is to curate the high-quality fine-tuning data to align Panacea to clinical trial vocabulary and create the specific instruction data for clinical trial tasks. Panacea consists of two main steps: an alignment step, which adapts Panacea to the vocabulary used in clinical trials, and an instruction-tuning step, which instructs Panacea on each clinical trial task. We built two datasets TrialAlign and TrialInstruct for the alignment step and the instruction-tuning step, respectively.

TrialAlign consists of 793,279 de-identified trial documents collected from 14 diverse sources and 1,113,207 scientific papers related to clinical trials (see Methods), representing a large-scale collection of clinical trial documents. By classifying these trial documents to terms in the International Classification of Diseases (ICD-10) ontology, we found that at least 100 conditions have 10,000 documents (Fig. 1a), indicating the good coverage of our dataset. Likewise, by classifying trial-related scientific papers to Medical Subject Headings (MeSH) terms, we found that at least 119 terms have more than 10,000 papers and at least 1,921 terms have more than 1,000 papers (Fig. 1b). The scale and the coverage of TrialAlign enable Panacea to be generalized to various conditions and treatments.

TrialInstruct contains instruction-tuning data from eight diverse tasks, including criteria design, study arm design, outcome measure design, patient-trial matching, query generation, query expansion, single-trial summarization, and multi-trial summarization, instructing Panacea on solving these tasks (Fig. 1c). Each task contains at least 2,000 data points, where each data point contains an instruction, an input, and an output (Fig. 1d). Since these eight tasks are related, we jointly fine-tuned the model using instruction data from these eight tasks, transforming Panacea into an all-in-one tool for clinical trial applications (Fig. 1e).

To evaluate Panacea, we built the first large-scale benchmark TrialPanorama that covers eight specific tasks in clinical trials (Table 1). Since these tasks contain both classification and generation tasks, TrialPanorama allows us to evaluate Panacea in various machine learning settings. We made this benchmark fully open-source.

Clinical trial search is an important task for clinical trial design and research. Trial designers often need to study similar trials to ensure their design aligns with existing trials. The goal of the trial search is to find relevant trials based on user inputs, which serves as the foundation for designing and matching trials. The key to a successful trial search is to create comprehensive search terms. As a result, we evaluate query generation, which converts unstructured user input to a list of keywords (Fig. 2a), and query expansion, which further expands this keyword list to relevant terms (Fig. 2b). These two tasks assess the ability to derive high-quality queries based on user intent, which is crucial for a successful trial search.

We first evaluated query generation by formulating it as a text classification problem that classifies user inputs into specific diseases, interventions, phases, status, and study types. We found that Panacea substantially outperformed existing approaches regarding the Jaccard index (Fig. 2d). The improvement is larger on diseases and interventions, which are more challenging due to the large number of classes in these two categories (Fig. 2c), indicating that Panacea can accurately convert user inputs into the structured format that is compatible with downstream machine learning classifiers.

Next, we evaluated query expansion by formulating it as a text generation problem. We did not provide the candidate keywords to the models since real-world keywords might have never been seen in the training trials. Similar to our observations in the query generation, Panacea achieved the best results on query expansion in terms of Jaccard index (Fig. 2e). We attribute the inferior performance of existing models on query expansion to the lack of fine-tuning on trial-related datasets. In contrast, Panacea is fine-tuned on TrialAlign, adapting it to the vocabulary used in clinical trials. The promising results of Panacea on query expansion and generation demonstrate its ability to precisely understand user intent, providing an accurate tool for finding relevant clinical trials.

Once similar trials are identified, the next task is to understand those trials via summarization. We evaluated the performance of Panacea on trial summarization. We studied both single-trial summarization, which aims to provide a concise summary of a specific trial study (Fig. 3a), and multi-trial summarization, which aims to summarize multiple trial studies that study similar conditions and interventions (Fig. 3b).

Since it could be biased to evaluate summarization using lexical-based metrics, we propose a novel metric based on large language models (see Methods, Supplementary Figures 1 and 2). In particular, we provided the ground truth summarization and the model-generated summarization to Claude and asked if these summarizations studied the same problem and made the same conclusion. We found that Panacea and comparison approaches can correctly summarize the trial goal, while the summarization of the trial conclusion is less accurate (Fig. 3c-d). Moreover, summarizing multiple trials is more challenging than summarizing a single trial based on the proposed metric. Nevertheless, our method still outperformed comparison approaches in summarizing multiple trials, suggesting its potential to assist researchers in extracting key information from many related trial studies.

We further used query generation and query expansion to evaluate trial summarization by extracting diseases, and interventions, and expanding them (Fig. 3c-d) from each trial. We examined whether the generated summarization can derive the same keywords as the ground truth summarization. We found that Panacea achieved the best performance on three of the six keyword categories while achieving comparable on the other categories. Moreover, we calculated the ROUGE score, which is used as the metric for trial summarization in previous works [25, 26], and observed improved performance by Panacea as well on multi-trial summarization (Fig. 3e). Finally, we used a case study to show that Panacea can correctly summarize the goal and the conclusion for 11 trial studies, while comparison models failed to (Fig. 3f).

The first step toward a successful trial execution is designing a detailed trial protocol synopsis. We evaluated Panacea on three tasks in trial design (See examples in Fig. 4a): Criteria design defines the eligibility criteria (i.e., the inclusion and exclusion criteria) for patient recruitment; Study arm design outlines the different treatment arms that will be applied to different patient subgroups; Outcome measures design specifies the metrics that are used to assess the trial success. We formulated these three tasks as a conditional text generation problem, which takes conditions, treatments, and the design of previous steps (e.g., reference criteria are used to generate study arms) as inputs to generate specific design text.

Because trials are described in plain text, we first exploited standard natural language processing metrics BLEU and ROUGE to evaluate the lexical similarity. We found that Panacea attained the best performance on all three clinical trial design tasks in terms of BLEU and ROUGE (Fig. 4b). First, we observed that Panacea substantially outperformed general-domain models, including our base model Mistral [27], confirming the benefit of fine-tuning using clinical trial-related data. Second, we found that Panacea improved the study arm design more than the other two tasks. Compared to criteria and outcome measures, study arm descriptions are more customized according to the disease and the treatment. The larger improvement of Panacea on study arms design demonstrates Panacea’s strong generalization ability. Finally, BioMistral [28], which is fine-tuned on general biomedical data, also outperformed Mistral, further demonstrating the value of domain-specific data. Nevertheless, Panacea still outperformed BioMistral by fine-tuning using our clinical trial-specific data TrialAlign and TrialInstruct, suggesting that data with improving domain specificity leads to better performance.

Lexical similarity metrics are widely used to evaluate text generation problems, but might not be clinically specific enough to evaluate the generations by Panacea. Recently, LLMs have been used to evaluate the generated text by exploiting their strong ability in text understanding. Here, we exploit Claude [29] to evaluate these three tasks by asking the model whether the generated task is clinically relevant (see Methods, Supplementary Figures 3-5). We found that Panacea outperforms all methods on criteria and study arms design, demonstrating the high quality of generation by Panacea (Fig. 4b).

Moreover, we examined a De Novo generation setting, using the generated output in the previous step as the input for the next step. For example, we used the generated criteria instead of the reference criteria as the input for generating study arms. De Novo generation frees users from providing any descriptions for the trial. We found that the performance of all methods dropped in this setting compared to the setting that utilizes reference input (Fig. 4c). Nevertheless, our method still outperforms all existing methods by a large margin, indicating its superior performance on this De Novo trial design. We further compared the generated text by three methods with the ground truth text on criteria design, where only Panacea can generate the correct criteria (Fig. 4d). Collectively, the promising performance of Panacea demonstrates its potential to automate clinical trial design.

We next evaluate the performance of Panacea on patient-trial matching. Given a patient note and a trial description, we aim to determine whether this patient is eligible for the trial by formulating this problem as a three-class classification task: eligible, excluded, or irrelevant (Fig. 5a).

We first evaluated our method on the TREC 2021 dataset [30], which consists of a training set and a test set. We used the training set to construct instructions in TrialAlign, and then assessed the performance of Panacea on the test set. We found that Panacea outperformed all comparison approaches in terms of balanced accuracy (BACC), Cohen’s KAPPA score, Recall, Precision, and F1, indicating the effectiveness of using TrialInstruct to fine-tune the model (Fig. 5b-f). To investigate the generalizability of our method, we further tested our method on the SIGIR dataset [31] where the entire dataset is used as the test set. We found that our method again attained the best performance on all three metrics, demonstrating the strong generalizability of our method.

As the eligible class is crucial for patient-trial recruitment, we further examined a binary classification setting. In this setting, we grouped ”excluded” and ”irrelevant” into one category, and ”eligible” into the other in order to determine whether a patient is eligible for a trial. Our method outperformed all comparison approaches in terms of F1, precision, and recall, indicating its applicability to real-world trial recruitment (Fig. 5g-i). Finally, we used a case study to illustrate how our method successfully classified a patient as eligible by examining each criterion and coming to a conclusion based on their criteria (Fig. 5j). In contrast, LLaMA-2 [32] made an incorrect conclusion by hallucinating an exclusion criterion not stated in the trial description.

The aim of constructing TrialInstruct is to provide Panacea with the ability to follow human instructions, especially in clinical trial domains.

We built the first large-scale benchmark TrialPanorama, including eight tasks in clinical trials. The training and test data constructed in the previous section are viewed as the benchmark data. We evaluated the models on TrialPanorama to assess each model’s performance across different clinical trial tasks.

In this section, we detail the techniques in Panacea, including the alignment and instruction finetuning steps.

In the trial search experiments, we focused on optimizing Panacea for two tasks: query generation and query expansion (see Supplementary Figure 11). These two tasks are pivotal for enhancing the efficiency and precision of searches within large clinical trial databases.

Query generation in this context essentially functions as a Named Entity Recognition (NER) task where the model identifies and categorizes key pieces of information from the trial descriptions relevant to user queries. To facilitate the generation of structured queries in a JSON format, we employed a specialized tool called JsonFormer [50]. This tool is instrumental in guiding the model to generate content for each key in the JSON structure sequentially.

Once the JSON format is generated, it is automatically converted into a Search Expression using a rule-based system. The conversion rules are straightforward: within the same key, terms are combined using the OR operator, and between different keys, the terms are combined using the AND operator. This structured approach ensures that the generated queries are precise and align well with the syntactical requirements of the search engines used in clinical trial databases.

For the query expansion task, this process enhances the original query by adding semantically related terms, thereby broadening the search scope to include relevant trials that may not use the exact phrasing of the original query terms. Panacea was trained to suggest additional keywords based on the initial input terms. The model learned to recognize and predict related terms that could be associated with the initial query, expanding the search breadth effectively.

The experiments on trial summarization were designed to test Panacea’s capabilities in condensing complex clinical trial information into succinct summaries. This component of our research focused on two specific tasks: single-trial summarization and multi-trial summarization (see Supplementary Figure 12).

To evaluate summarization tasks, we propose a novel metric based on Claude 3. We use Claude 3 to decide whether the model-generated summarization and the ground truth summarization studied the same problem and made the same conclusion, following prompts in Supplementary Figure 1 and 2. Specifically, Claude 3 directly outputs the goal alignment results for each test sample. For conclusion consistency, we first use Claude to evaluate model-generated summaries and ground truth summaries, respectively. Then, we calculate the matching accuracy between the model-generated summarization and ground truth summarization.

In our experimental setup for evaluating the Panacea model’s capabilities in clinical trial design, we utilized a multi-turn conversation format for the test data. This format consists of sequential (user, chatbot) pairs, reflecting a realistic interaction scenario where the model, acting as a chatbot, responds to user queries about designing a trial. The initial three rounds usually provide essential background information related to the trial design, such as the trial’s objectives, target population, and key endpoints. These initial conversations set the stage for the more complex interactions that follow. Starting from the fourth round of conversation, the model is tasked with predicting the chatbot’s responses based on the cumulative conversation history, which tests the model’s ability to maintain context and continuity over successive interactions.

To ensure the reliability of the experimental results and prevent the propagation of errors through the conversation chain, a teaching forcing strategy was implemented: regardless of the model’s output in any given round, the subsequent round’s input incorporates the groundtruth from the previous rounds rather than the model-generated responses. This method allows the model to be evaluated on its ability to adhere closely to a scientifically valid trial design path without being influenced by potential errors in its previous outputs.

To assess the relevance between models’ designed trials and ground truth, we employ Claude 3 to calculate clinical relevance. Specifically, we input each model’s output and the ground truth into Claude 3 to determine the relevance of the information generated by the model compared to the ground truth. The inputs to Claude 3 for clinical relevance evaluation are detailed in Supplementary Figures 3, 4, and 5, respectively. When a model’s outputs are relevant to the ground truth, Claude will output a 1; otherwise, it outputs a 0. We then calculate the clinical relevance using the following formula:

Here, “Relevance scores” refer to the series of 1s and 0s output by Claude 3 for each comparison between a model’s output and the ground truth. $N$ is the total number of outputs evaluated. This proportion reflects the percentage of times the model’s output was deemed clinically accurate relative to the ground truth, quantifying the frequency at which the model produces clinically relevant information.

In the patient-trial matching experiments, we employed a distinctive approach to training the Panacea model, focusing not on utilizing the entirety of the training data but rather on a selected subset. Initially, all available training data was subjected to a filtering process with Claude 3 Haiku. This involved predicting responses for each instance in the training set. Only those instances where Claude 3 Haiku’s predictions were accurate were retained for further processing. The rationale was to ensure that the model was learning from correctly reasoned examples and that the training data was high quality. The responses generated by Claude 3 Haiku, which correctly matched the groundtruth data, were then used as the new training corpus for Panacea. This step was crucial because the standard training datasets for patient-trial matching typically include labels indicating eligible or excluded but lack a detailed reasoning process for these outcomes. By incorporating Claude 3 Haiku’s responses, which involve step-by-step reasoning based on the input data, we injected reasoning capabilities into Panacea during the training process. Through this innovative training approach, Panacea showed superior performance in patient-trial matching tasks. The ability to reason and logically process eligibility criteria translated into higher accuracy and reliability in matching patients to appropriate trials. The evaluation prompt for patient-trial matching can be seen in Supplementary Figure 10.

The patient-trial matching is a three-class classification task for both SIGIR and TREC2021 datasets. Three classes for SIGIR are: 0) Would not refer this patient for this clinical trial; 1) Would consider referring this patient to this clinical trial upon further investigation; and 2) Highly likely to refer this patient for this clinical trial, while TREC2021 has: 0) Excluded (patient meets inclusion criteria, but is excluded on the grounds of the trial’s exclusion criteria); 1) Not relevant (patient does not have sufficient information to qualify for the trial); and 2) Eligible (patient meets inclusion criteria and exclusion criteria do not apply).