Do Clinicians Know How to Prompt? The Need for Automatic Prompt Optimization Help in Clinical Note Generation

The forward pass utilizes an LLM to generate summaries ($\hat{y}$) for $h$ from section $s$ by passing in a generic user-provided prompt ($p_{0}$), doctor-patient dialogue ($x$), and $s$. We use black box LLMs via API, denoted as $LLMp(i)$ ⁴⁴4$i$ is defined as all the inputs to the prompt (dialogue, section, etc.).. This API yields a probable text continuation, symbolized as $\hat{y}$, given a prompt. This prompt is a fusion of $p$ and $i$. Mathematically, $LLMp(i)$ is approximated by $\text{argmax}_{\hat{y}\in L}P_{\text{LLM}}(\hat{y}|p,i)$, where it selects the most likely continuation $\hat{y}$ from the set of natural language tokens $L$. The ones used for our method are OpenAI’s GPT3.5 and GPT4 ⁵⁵5We use OpenAI’s gpt-3.5-turbo-0613 and gpt-4-0613 in our experiments..

$p_{0}$ is a generic prompt such as the one shown in Figure 2 or Appendix A.4 that, in our use case, would be provided by a medical professional such as a clinician. It is a prompt that only instructs the model, in this step LLM $a$. $p_{0}$ and $x$ are passed into $a$ to output a generated summary $\hat{y}$. This first $\hat{y}$ is likely to be very suboptimal for $s$.

This segment of the algorithm represents the key transformational stage. The backward pass consists of (1) utilizing the same or a different LLM as before to provide suggestions on what is wrong with $\hat{y}$, (2) utilizing the LLM in step 1 to fix $p_{0}$ using the suggestions provided in step 1. Step 1 generates “gradients” and step 2 performs “backpropagation”.

The backward pass starts by passing in a fixed prompt ($p_{\nabla}$), $p_{0}$, $x$, $s$, the ground truth summaries ($y$), and $\hat{y}$ into an LLM $b$ to generate suggestions ($g$) on how to fix $p_{0}$ to make it more suitable for generating summaries for $s$. An example is shown in Appendix A.4. These suggestions are named “gradients”, the reason $p$ is labeled with $\nabla$. Note that $a\stackrel{{\scriptstyle?}}{{=}}b$, i.e. $a$ may or may not be equal to $b$.

Next, a fixed prompt ($p_{\delta}$), like the one shown in Appendix A.4, commands $b$ to use $g$ to fix $p_{0}$. $g$, $p_{0}$, and $p_{\delta}$ are passed into $b$. “gradient descent” happens here. $p_{\delta}$ resembles differentiation in traditional neural network training by using $g$ (the “gradient”) to guide the model toward a lower “loss”. Hence the $p$ is labeled with $\delta$. A new prompt $p^{\prime}$ is outputted by $b$, which should be closer to the optimal prompt $p^{*}$. $p^{*}=\text{argmax}_{p\in L}\{m(p,T)\}$, where $m(\cdot)$ represents a metric function and $T$ is all the training data for $s$. $p^{\prime}$ should be an edited version of $p_{0}$ that is in the opposite semantic direction.

At this point in the algorithm, the same $h$ is summarized again using $a$, but this time with $p^{\prime}$. The new summaries are evaluated against $y$.

$p^{\prime}$ is set to $p_{0}$ and the “iteration” restarts, repeating $j$ times. After $j$ iterations, the “epoch” is finished, and the final prompt, $p^{\prime}_{final}$, is used to generate summaries for a validation dataset $E$. These summaries are evaluated against $y$ to check the performance of $p^{\prime}_{final}$. The epochs are repeated $k$ times.

Enhancing the APO framework, we incorporate a human-in-the-loop component for prompt refinement. Post-APO, medical experts and laypersons review and adjust $p^{\prime}_{final}$ for each $s$, adding clinical acumen to the AI’s output. These revised prompts, $p^{\prime}_{final-human}$, are then evaluated by generating new summaries and scoring them against ground truths. The goal is to determine if there is a potential for human-AI collaboration on this task, and whether it should be with experts or not.

With 1.7k total doctor-patient dialogues and summaries, MTS-Dialog supports advances in automatic clinical note generation Abacha et al. (2023b, a). For our initial exploration of which GPT variants are the best across most sections (more details in Section 4.4), we use the original evaluation split of 100 data points. For APO, since the evaluation split is small, we merge the training and evaluation data into a single pool. The data is comprised of 20 SOAP sections. We discard sections with less than 10 data points, resulting in 14 sections that meet the criteria for further experimentation. Then, we randomly sample 5 data points from each section as training data. Detailed data distribution for these sections is outlined in the Appendix Table 3.

Models are evaluated with full-length F1-scores of ROUGE Lin (2004) and METEOR Banerjee and Lavie (2005). We use QuickUMLS⁶⁶6https://github.com/Georgetown-IR-Lab/QuickUMLS to extract medical concepts from both model-generated and ground truth summaries and then calculate F1-scores for these two lists of concepts, which is named UMLS-F1 Adams et al. (2023). We also add human preferences in Experiment Set-2.

We put the details of our dataset in Appendix 4.1. First, we designed the experiment to use the generic prompt, outlined in Appendix A.4, on six different GPT models ⁷⁷7text-ada-001, text-babbage-001, text-curie-001, text-davinci-003, gpt-3.5-turbo-0613, and gpt-4-0613. This objective was to evaluate which variants are the best across most sections, thereby guiding our selection for use in APO. We then divided our experiments into two sets ⁸⁸8After we got the different sets’ prompts, we then ran gpt-3.5-turbo-0613 or gpt-4-0613 API with self-consistency and zero-shot settings Wang et al. (2022), where temperature=0.3, run numbers=5. We used the default numbers for all other parameters in OpenAI API.:

Set-1: Comparative Analysis of APO and Human Contributions in Clinical Note Generation. This experiment aims to assess how APO, compared with humans, can assist in improving content generation for different sections of clinical notes. Specifically, we introduce a generic prompt and training data for distinct sections. The goal is to aid AI systems, such as GPT3.5 and GPT4, in identifying suitable section prompts that enhance content generation in each section. Our experiment involves four groups of prompters: medical experts ⁹⁹9One licensed physician, non-medical experts ¹⁰¹⁰10One has a master’s degree, and one has a bachelor’s degree. They do not have any medical background., GPT3.5 (with APO), and GPT4 (with APO). Each group modifies the generic prompt based on the training data for each section. We then compare the effectiveness of these modified prompts in assisting AI to generate summaries for different sections, using the results of the generic prompt as a baseline.

Set-2: Enhancing AI-Generated Clinical Content through Humans Prompt Modification Post-APO. In this set of experiments, we take the results of GPT3.5 (with APO) and GPT4 (with APO) as new baselines and invite medical experts and non-medical experts to further modify the prompts based on their knowledge and preferences. This approach examines how human intervention, post-APO implementation, affects the quality of AI-generated content in various clinical note sections. We analyze the effectiveness of these modifications by comparing them against the baseline established by APO-modified prompts, focusing on the nuances introduced by the domain-specific knowledge and preferences of the two human groups.

For our initial experiment, the findings indicate that GPT-4 and GPT3.5 emerged as the most effective variants, in descending order of performance, as detailed in Appendix A.5. As a result, they were used for our proposed algorithm.

Set-1: Comparative Analysis of APO and Human Contributions in Clinical Note Generation. Upon examining the ‘X guides GPT3.5’ results from Table 1 ¹¹¹¹11The details can be found in Appendix Table 5, we observed that expert and non-expert modifications resulted in slight improvements compared to the generic (baseline) results. However, according to the ROUGE and METEOR scores, ‘expert guides GPT3.5’ did not yield better outcomes than ‘non-expert guides GPT3.5’; non-experts led regarding factuality (UMLS-f1) scores. The performance of APO-GPT3.5 did not significantly differ from the baseline, whereas APO-GPT4 markedly surpassed all other methods. Compared to human modifications, APO-GPT4 enhanced summary quality, a feat APO-GPT3.5 did not achieve. For the same Table 1 ‘X guides GPT3.5’ experiment, the results indicated that prompts modified by experts, non-experts, and APO-GPT3.5 all fell short of the generic prompt across various sections, with expert modifications slightly outperforming non-experts, and both human groups surpassing APO-GPT3.5, especially in terms of factuality score. Consistent with the ‘X guides GPT3.5’ findings, APO-GPT4 again significantly elevated the scores across the board. Finally, the results in the Appendix Table 5 show the helpful effect of APO-GPT4 on problem (2) and (3) in Figure 1. These results further demonstrate GPT4’s emergent abilities in self-critique Madaan et al. (2023), self-feedback Huang et al. (2022), and self-explanation Zhao et al. (2023).

Set-2: Enhancing AI-Generated Clinical Content through Humans Prompt Modification Post-APO. In this experiment, we continued to explore the outcomes of the human-in-the-loop paradigm on top of APO. From the previous experiments in Table 1, it was evident that APO-GPT4 significantly boosted the summary quality, raising the lower bound of AI performance on this task and providing a new baseline for users to engage in further prompt engineering. We refer to the process of experts post-editing APO-refined APO-GPT4 prompts as ‘Exp-APO’ and the analogous post-editing by non-experts as ‘NoExp-APO’. We compared Exp-APO and NoExp-APO modifications, with the term ‘APO’ now exclusively referring to the results achieved by APO-GPT4. In Table 2, we found that for both ‘X guides GPT3.5’ and ‘X guides GPT4’, Exp-APO modifications did not significantly differ from APO-GPT4 in terms of ROUGE, METEOR, and UMLS-f1 scores, whereas NoExp-APO modifications notably degraded summary quality, particularly factuality scores, suggesting a loss of key information or the introduction of hallucinations.

In a detailed comparison between Exp-APO and APO-GPT4, we curated a human evaluation dataset from 100 randomly selected instances within the evaluation set. This allowed experts who contributed to Exp-APO to assess and provide feedback on their preference for summaries generated from their revised prompts compared to those produced by the original APO-GPT4 prompts. The outcome showed a preference distribution where 75% favored Exp-APO, 3% indicated no preference, and 22% preferred APO-GPT4. These results show that while factuality scores remained closely comparable, there was a slight decrease in ROUGE scores for Exp-APO, yet the expert preference was markedly in favor of Exp-APO. This can be attributed to how APO tends to enforce certain structural elements within prompts, such as explicitly stating ‘None’ in the absence of information. Experts tended to remove such repetitive formulations, which, although potentially reducing the strict adherence to format and the ROUGE score, did not impact the factuality score. Moreover, experts’ preferences are less influenced by rigid formatting and more by their own knowledge and experience. These expert insights, incorporated through the human-in-the-loop approach, may have introduced a degree of personalization to the prompts, aligning the AI-generated content more closely with human evaluative criteria and contributing to the overall preference for Exp-APO. This suggests that while expert post-editing prompts may not markedly enhance the quality of APO-GPT4 summaries, they align more closely with user preferences, offering a more personalized result without sacrificing summary quality.