Taxonomy-based prompt engineering to generate synthetic drug-related patient portal messages

Natalie Wang; Sukrit Treewaree; Ayah Zirikly; Yuzhi L Lu; Michelle H Nguyen; Bhavik Agarwal; Jash Shah; James Michael Stevenson; Casey Overby Taylor

doi:10.1016/j.jbi.2024.104752

Taxonomy-based prompt engineering to generate synthetic drug-related patient portal messages

J Biomed Inform. 2024 Dec:160:104752. doi: 10.1016/j.jbi.2024.104752. Epub 2024 Nov 25.

Authors

Natalie Wang¹, Sukrit Treewaree², Ayah Zirikly³, Yuzhi L Lu⁴, Michelle H Nguyen⁴, Bhavik Agarwal⁵, Jash Shah⁵, James Michael Stevenson⁶, Casey Overby Taylor⁷

Affiliations

¹ Department of Computer Science, Johns Hopkins University, Baltimore, 21218, MD, USA; Institute for Computational Medicine, Johns Hopkins University, Baltimore, 21218, MD, USA. Electronic address: [email protected].
² Institute for Computational Medicine, Johns Hopkins University, Baltimore, 21218, MD, USA; Department of Biomedical Engineering, Johns Hopkins University, Baltimore, 21218, MD, USA; Department of Medicine, Faculty of Medicine, Siriraj Hospital, Mahidol University, Bangkok, 10700, Thailand.
³ Center for Language and Speech Processing, Johns Hopkins University, Baltimore, 21218, MD, USA; Malone Center for Engineering in Healthcare, Johns Hopkins University, Baltimore, 21218, MD, USA.
⁴ Institute for Computational Medicine, Johns Hopkins University, Baltimore, 21218, MD, USA; Department of Biomedical Engineering, Johns Hopkins University, Baltimore, 21218, MD, USA.
⁵ Department of Computer Science, Johns Hopkins University, Baltimore, 21218, MD, USA; Institute for Computational Medicine, Johns Hopkins University, Baltimore, 21218, MD, USA.
⁶ Division of Clinical Pharmacology, Department of Medicine, Johns Hopkins University School of Medicine, Baltimore, 21205, MD, USA; Department of Pharmacology and Molecular Sciences, Johns Hopkins University School of Medicine, Baltimore, 21205, MD, USA.
⁷ Institute for Computational Medicine, Johns Hopkins University, Baltimore, 21218, MD, USA; Department of Biomedical Engineering, Johns Hopkins University, Baltimore, 21218, MD, USA; Division of General Internal Medicine, Department of Medicine, Johns Hopkins University School of Medicine, Baltimore, 21205, MD, USA.

PMID: 39603549
DOI: 10.1016/j.jbi.2024.104752

Abstract

Objective: The objectives of this study were to: (1) create a corpus of synthetic drug-related patient portal messages to address the current lack of publicly available datasets for model development, (2) assess differences in language used and linguistics among the synthetic patient portal messages, and (3) assess the accuracy of patient-reported drug side effects for different racial groups.

Methods: We leveraged a taxonomy for patient- and clinician-generated content to guide prompt engineering for synthetic drug-related patient portal messages. We generated two groups of messages: the first group (200 messages) used a subset of the taxonomy relevant to a broad range of drug-related messages and the second group (250 messages) used a subset of the taxonomy relevant to a narrow range of messages focused on side effects. Prompts also include one of five racial groups. Next, we assessed linguistic characteristics among message parts (subject, beginning, body, ending) across different prompt specifications (urgency, patient portal taxa, race). We also assessed the performance and frequency of patient-reported side effects across different racial groups and compared to data present in a real world data source (SIDER).

Results: The study generated 450 synthetic patient portal messages, and we assessed linguistic patterns, accuracy of drug-side effect pairs, frequency of pairs compared to real world data. Linguistic analysis revealed variations in language usage and politeness and analysis of positive predictive values identified differences in symptoms reported based on urgency levels and racial groups in the prompt. We also found that low incident SIDER drug-side effect pairs were observed less frequently in our dataset.

Conclusion: This study demonstrates the potential of synthetic patient portal messages as a valuable resource for healthcare research. After creating a corpus of synthetic drug-related patient portal messages, we identified significant language differences and provided evidence that drug-side effect pairs observed in messages are comparable to what is expected in real world settings.

Keywords: Biomedical informatics; ChatGPT; Large language models; Natural language processing; Patient portal messages; Synthetic data.

MeSH terms

Drug-Related Side Effects and Adverse Reactions
Electronic Health Records
Humans
Language
Linguistics
Patient Portals*
Racial Groups