MEDFuse: Multimodal EHR Data Fusion with Masked Lab-Test Modeling and Large Language Models

Most recent works in medical Multi-Label Text Classification (MLTC) entirely rely on medical texts. For instance, Kim et al. (Chen, 2015) introduced a convolutional attention network designed to extract meaningful document representations across varying text lengths. Recent developments in LLMs, such as those discussed by Luo et al. (2022) and Elliot et al. (Bolton et al., 2024), utilize extensive data from medical literature for domain-specific tasks such as natural language inferencing. Additionally, some studies have employed graph neural networks (GNNs) to organize sequences from Electronic Medical Records (EMR) into hierarchical graphs (Wu et al., 2021), or to integrate entity relationships from text using attention mechanisms in neural networks (Chen et al., 2019; Dun et al., 2021). Nevertheless, many of these studies overlook the potential advantages of integrating medical expert knowledge from official guidelines and critical blood tests. A combined approach that harnesses both unstructured and structured data could offer extra help to offset issues like label and data scarcity in the medical domain.

Recent work has leveraged self-supervised learning methods, like contrastive pretraining of clinical notes (Cai et al., 2024) and prompt-based large language modeling (Ding et al., 2024; Hegselmann et al., 2023), to facilitate multimodal learning of EHR data. The former encourages the alignment between paired patient data via contrastive loss, and the latter usually directly converts the structured data into text by prompt templates and feeds it into LLMs. However, if the data fusion process focuses solely on aligning the common information, such as diabetes history (text) and blood glucose levels (lab), the rich, modality-specific insights like exercise habits may be overlooked. This can limit understanding of the patient’s health and impact predictive models and clinical decisions. Therefore, it is essential to develop multimodal EHR data fusion techniques that can effectively capture and integrate both modality-specific and modality-shared information.

Given the clinical notes and lab tests of the patient’s current and historical visits, MEDFuse integrates clinical notes and lab test data to create a comprehensive patient representation for accurate multi-disease prediction. Firstly, as illustrated in Figure 1, in the Multimodal Embedding Extraction stage, textual data from clinical notes, including detailed patient information and medical history, are filtered and structured. Simultaneously, abnormal numerical data from various lab tests, such as triglyceride levels, HDL cholesterol, ALT (SGPT), and glucose levels, are extracted and formatted into textual data by prompt templates. The filtered clinical text is then processed by fine-tuned LLMs to generate embeddings that capture its semantic meaning. In parallel, the raw structured tabular data are processed using a domain-specific masked lab-test model to create embeddings representing the quantitative lab data. The two embeddings are then passed through the disentangled transformer module for multimodal fusion and final disease prediction.

Initially, features from each modality are multiplied by the Kronecker product to approximate a joint distribution, $C=A\bigotimes B\in\mathbb{R}^{(m\times(a)\times(b))}$, effectively capturing the pairwise interactions. Self-attention is applied to $Z_{a}$ and $Z_{b}$ to obtain $S_{a}$ and $S_{b}$, controlling the expressivity of each modality and preventing noisy features. Subsequently, the common information of the joint distribution is extracted via cross attention of $Q_{c}$, $K_{c}+K_{a}+K_{b}$, and $V_{c}+V_{a}+V_{b}$ to model modality-common features $S_{c}$. To preserve modality-specific information, we minimize the Mutual Information (MI) loss between concatenated $S_{a}+S_{b}$ and $S_{c}$. As the computation of mutual information is intractable, we calculate a variational upper bound called contrastive log-ratio upper bound (vCLUB) as an MI estimator to achieve MI minimization. Given two variables $a$ and $b$, the $L_{v}^{CLUB}(a,b)$ is calculated as follows (Zhang et al., 2024):

We employ an MLP $q_{\theta}(b|a)$ to provide a variational approximation of $q_{\theta}(b|a)$, which can be optimized by maximizing the log-likelihood (Zhang et al., 2024): $L_{\text{estimator}}(a,b)=\frac{1}{N}\sum_{i=1}^{N}\log q_{\theta}(b_{i}|a_{i})$. The MI Loss is then calculated as: $\text{MI Loss}=L_{v}^{CLUB}(S_{a}+S_{b})+L_{\text{estimator}}(S_{a}+S_{b},S_{c})$.

After optimizing the mutual information between the modality-specific information and the modality-common information, we utilize dense fusion (Holste et al., 2023) to enable denser interaction between modalities. Instead of directly connecting a prediction classifier on top of the fused representation $S_{c}$, we learn deeper representations of the clinical notes and lab test features and add skip connections to concatenate with the fused representation, forming a final fused embedding: $h_{a}=f_{a}(S_{a})\text{ and }h_{b}=f_{b}(S_{b})$ where $f_{a}$ and $f_{b}$ are fully-connected layers. This final representation not only aggregates the modality-specific features but also incorporates the modality-common representation from the previous stage of the network: $h_{final}=concat(h_{a},S_{c},h_{b})$. Finally, a dense block $g$ is used to generate $y=g(h_{final})$, and the model is trained by optimizing the prediction loss (focal loss for multilabel prediction). This allows for dense interaction of features from each modality, aggregating information across different stages of the network. The final loss optimizes a combination of the prediction objective and the mutual information loss, controlled by a hyperparameter $\lambda$ with a value range of [0,1]. In this case, we choose a value of 0.1. $\text{Loss}_{\text{final}}=L_{\text{objective}}(g(h_{\text{final}}))+\lambda*\text{MI}(\text{concat}(S_{a},S_{b}),S_{c})$

To evaluate the performance of the methods under comparison, we employed two real-world EHR datasets: MIMIC-III (Johnson et al., 2016) and FEMH. We collected five years of EHRs from the Far Eastern Memorial Hospital (FEMH) in Taiwan from 2017 to 2021. The dataset includes 1,420,596 clinical notes, 387,392 lab results, and over 1,505 lab test items. The FEMH Research Ethics Review Committee ¹¹1https://www.femhirb.org/ approved the study, and all data were de-identified. We selected patients with at least two recorded visits from each dataset.

For the multi-label classification task in MIMIC-III, we identified the top 10 most prevalent conditions: “Hypertension, uncomplicated”, “Cardiac arrhythmias”, “Fluid and electrolyte disorders”, “Congestive heart failure”, “Diabetes w/o chronic complications”, “Chronic pulmonary disease”, “Valvular disease”, “Renal failure”, “Hypertension, complicated”, and “Other neurological disorders”. In the FEMH dataset, the top 10 most common diseases include “Hypertension”, “Diabetes”, “Heart disease”, “Cancer”, “Cerebrovascular Disease”, “Kidney Disease”, “Liver Disease”, “Asthma”, “Hyperlipidemia”, and “Lung Disease”. We applied several established multi-label classification metrics to assess model performance such as Macro-average and weighted-average F1-Scores, precision, recall, and accuracy on the test dataset (Lipton et al., 2014; Hossin and Sulaiman, 2015; Palacio-Niño and Berzal, 2019).

Table 1 and Table 2 illustrate the training and validation performance of various models, highlighting the effectiveness of our proposed method on the MIMIC-III and FEMH datasets, respectively. In Table 1, our approach outperforms baseline models such as Bert (Devlin et al., 2018), Mistral-7B-v0.1 (Jiang et al., 2023), Llama-2-7B-hf (Touvron et al., 2023), Meta-Llama2-13B (AI@Meta, 2024a), Meta-Llama3-8B (AI@Meta, 2024b), and Medical-Llama3-8B (ruslanmv, 2024) across all key metrics. Specifically, MEDFuse shows significant improvements over the best-performing LoRA fine-tuned LLM, Medical-Llama3-8B. On the test set, our model performs 1.49% better in macro F1 score, and similar trends are observed in other metrics. Table 2 shows MEDFuse consistently outperforms Medical-Llama3-8B on the FEMH dataset. For example, training and validation in precision is a 1.53% increase, in the recall is a 2.07% increase, the training accuracy is 0.9311 (0.55% increase), and validation accuracy is 0.9296 (0.41% increase). These results validate the robustness and generalizability of our approach, underscoring its potential for accurate and reliable clinical predictions across diverse datasets.

We conducted an ablation study to examine the contributions of various components in our proposed method, which integrates Medical-Llama3 with a transformer module, utilizing lab tests (LABTEXT) and clinical notes (TEXT). The results highlight clear performance contrast when any component is omitted. Removing both the transformer and LABTEXT results in a 4.81% drop in training precision and a 4.40% decrease in validation precision. The most substantial performance reduction occurs when both the transformer and TEXT are excluded, leading to a 29.76% decrease in training precision and a 30.66% decrease in validation precision. This underscores the indispensable role of TEXT and the transformer in our method. Even when only TEXT is removed, performance significantly deteriorates, with a 17.14% decline in training precision and a 14.60% decline in validation precision. These findings illustrate that each component contributes significantly to the model’s overall efficacy. Our full model, combining LLMs and MLTM, demonstrates the highest performance, with a training accuracy of 0.9535 and a validation accuracy of 0.9122.