Query-Guided Self-Supervised Summarization of Nursing Notes

A patient typically has multiple nursing notes ordered in time to document the evolution of her condition. Therefore, the key information crucial for summarizing a patient’s current status is influenced by the context provided by the prior notes. We regard this as temporal information which should be incorporated during summarization to help the model understand the progression of the patient’s condition.

For $N_{i}$, the embeddings of its previous notes are represented by the embeddings of their respective first tokens, which are special tokens indicating the start of each note. These embeddings are obtained at the last hidden state in the $\operatorname{ENC}$, denoted as $\{\mathbf{h_{1}},\mathbf{h_{2}},...,\mathbf{h_{i-1}}\}$, where $\mathbf{h_{i}}\in\mathbb{R}^{d}$ and $d$ is the dimension of the hidden state. We aggregate the representations of the past notes by weighted mean pooling such that the most recent notes receive the largest weight. In practice, we determine initial weights $\beta_{j},j=1,\ldots,i-1$ for each past note $N_{i}$ based on the position in the sequence, such that $\beta_{1}=1,\beta_{2}=2,\ldots,\beta_{i-1}=i-1$. The weighted mean pooling is performed using normalized weights:

where $\mathbf{h}^{TIF}\in\mathbb{R}^{d}$ represents the information fusion of the past notes. As shown in Figure 3, we prepend a special token [TI] at the beginning of the decoder input, representing temporal information with embedding $\mathbf{h}^{TIF}$. Consequently, the initial input to the decoder at the first time step consists of [[TI], [BOS]], where [BOS] is a special token indicating the start of generation. We substitute [TI] with the padding token [PAD] for nursing notes that have no past notes.

The model generates subsequent tokens in the summary in an auto-regressive manner. At each time step, the token produced is appended at the end of the decoder input for generation of subsequent tokens. Therefore, the [TI] token contributes to the generation of each token in the summary, serving as a prompt which consistently guides the model to focus on information about the patient’s past condition.

We aim at obtaining summaries focusing on the patient’s condition. To aid this, we explicitly incorporate patient-level information into the model through a cross-attention mechanism, which facilitates the interaction of information on different levels of representation learning. A patient’s metadata PA typically comprise basic information recorded for the patient’s admission, including age, gender, existing diagnoses, and performed procedures. We convert this metadata into patient information in natural language (one example presented in Appendix A.1), and then encode it using $\operatorname{ENC}$ to derive patient embedding $\mathbf{H}^{PA}\in\mathbb{R}^{z\times d}$ for patient $PT$, where $z$ represents the number of tokens in patient information. The encoder also learns the embedding of the source note, $\mathbf{H}^{enc}\in\mathbb{R}^{n\times d}$, where $n$ denotes the number of tokens in the note given as input to the encoder. On the decoder $\operatorname{DEC}$ side, let us assume the tokens input to the decoder at the current timestep are [[TI], [BOS], $y_{1}$,…,$y_{j}$]. Consequently, the hidden representation passed to the $l$th decoder layer is $\mathbf{H}_{l}^{dec}\in\mathbb{R}^{{(j+2)}\times d}$. The hidden representation $\mathbf{H}_{l}^{dec}$ is processed and updated in each decoder layer using the conventional self-attention and cross-attention with $\mathbf{H}^{enc}$. Furthermore, we augment the decoder layer with patient information by performing cross-attention also between the hidden representation $\mathbf{H}_{l}^{dec}$ and the patient embedding $\mathbf{H}^{PA}$. This facilitates the fusion of patient- and note-level information:

where the $\operatorname{MHCA}$ and the $\operatorname{MHSA}$ respectively denote Multi-Head Cross-Attention and Multi-Head Self-Attention (Vaswani et al., 2017). $\lambda_{2}\in[0,1]$ is a hyperparameter to control the importance of patient meta information. $\mathbf{H}_{l+1}^{dec}$ is the input to the next decoder layer, or if the $l$th layer is the final layer, it is the input to the language modeling head.

With these two augmentation blocks, the computation of the final decoder state $\mathbf{H}^{dec}\in\mathbb{R}^{(j+2)\times d}$ for generating the $(j+1)$th token in the summary of the note $N_{i}$ is abstracted as:

$\operatorname{LMH}$ (Language Modeling Head) maps $\mathbf{H}^{dec}$ to a probability vector $\mathbf{v}\in\mathbb{R}^{vs}$ over the vocabulary of size $vs$. $\mathbf{v}$ is processed using the straight-though gumbel softmax trick, denoted as $\operatorname{ST-GumbelSoftmax}$, resulting in a one-hot vector $\mathbf{v^{\prime}}\in\mathbb{R}^{vs}$ providing the index of the $(j+1)$th token.

Given a nursing note pair $(N_{i},N^{\prime}_{i})$, we regard the query about whether $N^{\prime}_{i}$ is the successor note of $N_{i}$ as a prediction of the patient’s future status. To train the query responder R for the next note prediction, we create two note pairs for each nursing note, where the positive pair comprises the note and its successor in the sequence, and the negative pair contains the note and a randomly chosen non-consecutive note. If $N_{i}$ is the patient’s last nursing note, we use the patient’s discharge summary and a random note from other patients to construct the positive and negative pairs. The query is formulated as binary classification, and the output of $\operatorname{R}$ is the probability of each pair being the positive pair containing the consecutive notes.

Readmission information is easily retrieved from the hospital’s database and is closely related to the patient. The readmission prediction query is formulated as a 2-class classification task to predict whether the patient will be readmitted within 30 days of discharge, which reflects the patient’s future condition.

Classifying a patient’s diagnosis status or phenotype is a query to the patient’s current status. Following Harutyunyan et al. (2019), phenotype classification is formulated as a multi-label classification, where ICD-9 diagnosis codes mapped by HCUP CCS code groups¹¹1https://hcup-us.ahrq.gov/toolssoftware/ccs/ccsfactsheet.jsp are categorized into 25 classes. Therefore, the responder outputs the probability distribution of the phenotype as $[p_{1},p_{2},...,p_{25}]$.

We investigate the combined utilization of two queries, readmission prediction, and phenotype classification, to see if joint guidance is more effective. After obtaining the result of readmission prediction $[p^{r}_{1},p^{r}_{2}]$ and the result of phenotype classification $[p^{c}_{1},\ldots,p^{c}_{25}]$, we integrate them by converting the results into a 50-class probability distribution.

Metrics in the automatic evaluation are divided into three categories: 1) predictiveness, 2) factuality and consistency, and 3) conciseness.

Metrics for predictiveness assess whether the summary adequately contains patient key information, quantified as the ability to predict the patient’s condition using the summary as input. Specifically, we conduct readmission prediction and phenotype classification using summaries from baselines and our method. We employ the summaries generated by different methods to fine-tune the query responder, resulting in multiple predictors, one for each method. For readmission prediction, we report the weighted F1 and F1 of the positive class (“being readmitted”), and for phenotype classification, we report the F1-Macro score.

For consistency and factuality, we consider: (1) UMLS-Recall. This metric measures the biomedical information consistency by comparing the set of medical concepts in the summary with that in the original note. We employ QuickUMLS (Soldaini and Goharian, 2016) to extract Unified Medical Language System (UMLS) biomedical concepts from the nursing note and its summary. Recall is the proportion of concepts in the original note that are present in the summary. (2) UMLS-FDR. FDR denotes False Discovery Rate. Analogously to UMLS-Recall, we compute the proportion of medical concepts in the summary that do not appear in the original note. (3) FactKB. With FactKB (Feng et al., 2023), an evaluation model measuring the factuality of a summary and its original text, we evaluate whether the summaries are consistent with the nursing notes from the perspective of their overall semantic information. (4) BARTScore. This metric evaluates the general consistency of summaries in a text-generation manner using BART, which also considers aspects such as the structure, coherence, and fluency of the summary (Yuan et al., 2021).

Finally, we report the length of the generated summary as a percentage of the original note’s length to assess conciseness. We do not enforce a strict maximum length for baselines because we believe the model should be capable of determining the appropriate length autonomously.

Without a reference summary, automatic evaluation metrics may not fully capture the quality of the summary. Therefore, we invite a clinician to conduct manual evaluation of the summaries of 25 nursing notes. The clinician evaluates summaries from multiple methods in a blinded and randomized order. Each summary is rated on four aspects: (1) Informativeness: Whether the summary adequately captures essential information regarding the patient’s condition in the original note; (2) Fluency: Whether the summary is well-written and easy to understand. (3) Consistency: How well the summary aligns with the original nursing note in factuality. (4) Relevance: It evaluates the conciseness of the summary and whether it contains unnecessary information. The score for each aspect ranges from 1 to 5. More detailed grading criteria are presented in Appendix B.2.

Results are shown in Table 1. In the readmission prediction task, GPT-4 performs best, producing summaries that enable more accurate prediction of a patient’s status. Our method also exhibits excellent performance, surpassing all few-shot methods. The main reference for our method is BART-zs, as it is the base model in our method, and hence represents performance without the proposed novel components. We see that our method outperforms BART-zs significantly in weighted F1 score (84.2 vs. 78.8) and F1 score of the positive class (18.2 vs. 11.1). This shows the effectiveness of the adaptation strategy guiding the model with useful queries. Interestingly, we find that using the summary from GPT-4 for this task outperforms using the original notes. Similarly, the summary from our method has performance close to that of using the original notes, highlighting the importance of high-quality summaries. In phenotype classification, our method with the query focusing on patients’ phenotype performs the best, outperforming BART-zs in Macro F1 (25.6 vs. 20.5). Even when using the similarity alone as a guiding signal, our method still is better than BART-zs (22.4 vs. 20.5) or BART-fs (22.4 vs. 21.1). Although specialized in text summarization, Pegasus has weak performance on all predictiveness metrics.

As shown in Table 1, there is an expected trade-off between UMLS-Recall and the summary’s length. Our method strikes a good balance between medical information consistency (measured by UMLS-Recall) and conciseness. GPT-4 captures more medical information, but achieves this with summaries which are less concise. Conversely, BART-zs can produce concise summaries but fails to adequately capture medical concepts. Even if the 10-shot fine-tuning clearly improves predictiveness and UMLS-Recall, BART-fs still struggles to generate a concise summary. Similar to BART-fs, both BioMistral-zs and BioMistral-fs tend to produce summaries that are not concise.

Summaries generated from BART-zs maintain high levels in factuality (measured by UMLS-HR and FactKB) and general consistency (measured by BARTScore). Our method also has strong performance on relevant metrics. We find that although LLMs, such as GPT-4 and BioMistral, excel in language understanding, they do not perform well on factuality and general consistency. One possible reason is their tendency to rephrase or even expand upon the original notes, potentially introducing inconsistent information. However, the metrics can be influenced by text style and layout, which may cause summaries that are more different from the original note to score relatively lower, even if they are more fluent. For this reason, our model also scores lower than the base model BART-zs on some metrics. The results and limitations will be further discussed in Sections 5.4 and 5.5.

According to the results shown in Table 1, the performance with different queries varies. We can observe: (1) Regarding predictiveness, employing queries closely related to patients and focusing on readmission and phenotype information yield superior performance compared to other queries. As expected, the method with phenotype-related queries performs the best in phenotype classification, while the method with readmission-related queries is the best in readmission prediction. This highlights the effectiveness of guiding the summarization with queries, and different queries enable the summary to concentrate on different aspects of the original note. (2) Using similarity as guidance can produce summaries that are more similar to the original notes, resulting in higher scores on general consistency and factuality. However, summaries generated under this configuration tend to be longer and often sacrifice predictiveness and informativeness regarding medical concepts, demonstrating the limitations of the unconstrained guidance signal. (3) When employing joint guidance with both readmission and phenotype information, our method consistently achieves excellent performance across all metrics. This indicates that combining different guidance signals can help in producing better summaries, and further research is needed to explore this aspect in depth.

Our method significantly outperforms the base model BART-zs on all four metrics. This indicates that the proposed domain adaptation strategy enables the model to generate higher-quality summaries from the medical personnel’s perspective, containing refined and important patient information with fewer hallucinations and increased readability. Although on average the summaries generated by our method are longer than those produced by the base model, our method achieves a higher relevance score from the clinician, suggesting the base model struggles to identify key information and focuses on unnecessary details. Our model can effectively enhance this aspect.

Due to the LLMs’ strong language understanding capability and sufficient medical knowledge, GPT-4 and BioMistral-fs can adequately summarize key information in nursing notes, receiving approximately the same average score in Informativeness as our method. Additionally, they excel in generating fluent summaries by rephrasing and clarifying abbreviations, receiving a slightly higher Fluency from the clinician than our method. However, the rephrasing can introduce factual inconsistencies, and the tendency to infer additional content may reduce factuality. Consequently, our model has a higher average score in Consistency, which is essential in the clinical setting. Furthermore, it generates more concise summaries, leading to a higher average score in Relevance. However, due to the small sample size, the only statistically significant difference in these comparisons was the improvement of our method compared to Biomistral-fs in consistency, and further work is needed for conclusive results.

We calculated the significance in Figure 4 using a two-tailed Binomial test on the pairwise win-rates. In detail, we count the number of nursing notes where our method has a score higher or lower than a comparison method and test for the null hypothesis that the win-probability is 0.5.

A high-quality summary should facilitate efficient understanding of the relevant content for users. In the context of nursing notes, this means the summary should capture the patient’s condition. Our method employs patient-related queries, indirectly ensuring that the summary centers around the patient’s status. The summaries generated with different queries can be seen as coming from distinct conditional distributions and parts of the semantic space. As discussed in Section 5.1, the queries can guide summaries to focus on specific aspects of the original note. Therefore, by selecting appropriate queries, we can control preferences for desired content and adjust granularity, which facilitates a more flexible and user need -oriented summary generation. For instance, broad queries about the patient’s condition will result in a summary that focuses on the patient’s overall condition, while more detailed queries, such as those regarding cardiovascular health, could produce a summary that focuses on that specific aspect.

One challenge is how to efficiently integrate information into the model while avoiding excessive computational cost. We utilize cross-attention to allow the patient’s metadata to efficiently interact on multiple levels with the process of generating the summary. In contrast, for temporal information in previous notes, using cross-attention in a similar manner might make it difficult for the model to balance attention across the current note, past notes, and patient information, in addition to introducing computational challenges with long sequences of notes. Hence, we adopt a simple but effective strategy: representing the temporal information, obtained by weighted mean pooling from previous note representations, as the first token of the decoder’s input. This strategy is intuitive, as information from previous notes naturally precedes the summary of the current note.

The metrics used in the automatic evaluation have limitations as they do not conclusively reflect the quality of the summary, and come with trade-offs. For example, a good performance in predictiveness and medical information consistency (UMLS-Recall) may not be due to the high quality of the summary but rather caused by copying the source note, resulting in a lack of conciseness and fluency. Conversely, as the summary becomes more concise, it may become less informative. Furthermore, models used to measure factuality and general consistency have inherent biases. As they are based on general semantics, they are potentially weak at recognizing patient-related information due to the dissimilarity between their training domain and clinical data, and they often prioritize text style and structure. Finally, since BARTScore is derived from the BART model, summaries generated by BART have a bias of scoring relatively higher with this metric. We attempt to mitigate the impact of these limitations by comprehensively considering multiple metrics, and including the manual evaluation by a clinician, but there remains a need for more conclusive evaluation metrics.

(1) Our current approach produces summaries of individual nursing notes, lacking the long context and support for multiple note summarization. (2) There is room for more exploration on the formulation of the clinical queries. We don’t employ generative queries but only queries related to the classification of the patient’s status. Also, when investigating the combined effects of multiple queries, further exploration using multi-task learning methods could be beneficial. (3) Due to the workload, the number of notes assessed in the manual evaluation is limited to 25. A larger sample size would allow more conclusive comparisons of the strengths and weaknesses of the methods.

We presented a novel method for self-supervised nursing note summarization, where the main innovation was the introduction of query guidance, which successfully directed the summaries to include desired content. In the manual evaluation by a professional clinician, our method significantly outperformed a specialized open text summarization model, BART-Large-CNN, in all metrics. Because this model was the base model of our method, the result highlights the usefulness of the novel developments. Of the other baselines, the proprietary GPT-4 had the closest performance to our method and was better than the other baselines. In the automatic evaluation, GPT-4 was better than our method in predictiveness, but, importantly, our method outperformed GPT-4 in factual consistency, having fewer hallucinated facts without sacrificing the correct content. The same trend was seen in the manual evaluation as higher average consistency for our method, although the difference was not statistically significant. Hence, our approach can produce more reliable summaries, clearing obstacles for responsible clinical use of LLMs. From the machine learning perspective, our method demonstrates the feasibility of domain adaptation for pre-trained text summarization models without explicit supervision, and the effectiveness of self-supervised strategies to guide conditional summarization to specific interests.