Showing 1–7 of 7 results for author: Tso, G

Extrinsically-Focused Evaluation of Omissions in Medical Summarization

Authors: Elliot Schumacher, Daniel Rosenthal, Varun Nair, Luladay Price, Geoffrey Tso, Anitha Kannan

Abstract: The goal of automated summarization techniques (Paice, 1990; Kupiec et al, 1995) is to condense text by focusing on the most critical information. Generative large language models (LLMs) have shown to be robust summarizers, yet traditional metrics struggle to capture resulting performance (Goyal et al, 2022) in more powerful LLMs. In safety-critical domains such as medicine, more rigorous evaluati… ▽ More The goal of automated summarization techniques (Paice, 1990; Kupiec et al, 1995) is to condense text by focusing on the most critical information. Generative large language models (LLMs) have shown to be robust summarizers, yet traditional metrics struggle to capture resulting performance (Goyal et al, 2022) in more powerful LLMs. In safety-critical domains such as medicine, more rigorous evaluation is required, especially given the potential for LLMs to omit important information in the resulting summary. We propose MED-OMIT, a new omission benchmark for medical summarization. Given a doctor-patient conversation and a generated summary, MED-OMIT categorizes the chat into a set of facts and identifies which are omitted from the summary. We further propose to determine fact importance by simulating the impact of each fact on a downstream clinical task: differential diagnosis (DDx) generation. MED-OMIT leverages LLM prompt-based approaches which categorize the importance of facts and cluster them as supporting or negating evidence to the diagnosis. We evaluate MED-OMIT on a publicly-released dataset of patient-doctor conversations and find that MED-OMIT captures omissions better than alternative metrics. △ Less

Submitted 14 November, 2023; originally announced November 2023.

DERA: Enhancing Large Language Model Completions with Dialog-Enabled Resolving Agents

Authors: Varun Nair, Elliot Schumacher, Geoffrey Tso, Anitha Kannan

Abstract: Large language models (LLMs) have emerged as valuable tools for many natural language understanding tasks. In safety-critical applications such as healthcare, the utility of these models is governed by their ability to generate outputs that are factually accurate and complete. In this work, we present dialog-enabled resolving agents (DERA). DERA is a paradigm made possible by the increased convers… ▽ More Large language models (LLMs) have emerged as valuable tools for many natural language understanding tasks. In safety-critical applications such as healthcare, the utility of these models is governed by their ability to generate outputs that are factually accurate and complete. In this work, we present dialog-enabled resolving agents (DERA). DERA is a paradigm made possible by the increased conversational abilities of LLMs, namely GPT-4. It provides a simple, interpretable forum for models to communicate feedback and iteratively improve output. We frame our dialog as a discussion between two agent types - a Researcher, who processes information and identifies crucial problem components, and a Decider, who has the autonomy to integrate the Researcher's information and makes judgments on the final output. We test DERA against three clinically-focused tasks. For medical conversation summarization and care plan generation, DERA shows significant improvement over the base GPT-4 performance in both human expert preference evaluations and quantitative metrics. In a new finding, we also show that GPT-4's performance (70%) on an open-ended version of the MedQA question-answering (QA) dataset (Jin et al. 2021, USMLE) is well above the passing level (60%), with DERA showing similar performance. We release the open-ended MEDQA dataset at https://github.com/curai/curai-research/tree/main/DERA. △ Less

Submitted 29 March, 2023; originally announced March 2023.

Motif-aware temporal GCN for fraud detection in signed cryptocurrency trust networks

Authors: Song Li, Jiandong Zhou, Chong MO, Jin LI, Geoffrey K. F. Tso, Yuxing Tian

Abstract: Graph convolutional networks (GCNs) is a class of artificial neural networks for processing data that can be represented as graphs. Since financial transactions can naturally be constructed as graphs, GCNs are widely applied in the financial industry, especially for financial fraud detection. In this paper, we focus on fraud detection on cryptocurrency truct networks. In the literature, most works… ▽ More Graph convolutional networks (GCNs) is a class of artificial neural networks for processing data that can be represented as graphs. Since financial transactions can naturally be constructed as graphs, GCNs are widely applied in the financial industry, especially for financial fraud detection. In this paper, we focus on fraud detection on cryptocurrency truct networks. In the literature, most works focus on static networks. Whereas in this study, we consider the evolving nature of cryptocurrency networks, and use local structural as well as the balance theory to guide the training process. More specifically, we compute motif matrices to capture the local topological information, then use them in the GCN aggregation process. The generated embedding at each snapshot is a weighted average of embeddings within a time window, where the weights are learnable parameters. Since the trust networks is signed on each edge, balance theory is used to guide the training process. Experimental results on bitcoin-alpha and bitcoin-otc datasets show that the proposed model outperforms those in the literature. △ Less

Submitted 29 March, 2023; v1 submitted 21 November, 2022; originally announced November 2022.

COVID-19 in differential diagnosis of online symptom assessments

Authors: Anitha Kannan, Richard Chen, Vignesh Venkataraman, Geoffrey J. Tso, Xavier Amatriain

Abstract: The COVID-19 pandemic has magnified an already existing trend of people looking for healthcare solutions online. One class of solutions are symptom checkers, which have become very popular in the context of COVID-19. Traditional symptom checkers, however, are based on manually curated expert systems that are inflexible and hard to modify, especially in a quickly changing situation like the one we… ▽ More The COVID-19 pandemic has magnified an already existing trend of people looking for healthcare solutions online. One class of solutions are symptom checkers, which have become very popular in the context of COVID-19. Traditional symptom checkers, however, are based on manually curated expert systems that are inflexible and hard to modify, especially in a quickly changing situation like the one we are facing today. That is why all COVID-19 existing solutions are manual symptom checkers that can only estimate the probability of this disease and cannot contemplate alternative hypothesis or come up with a differential diagnosis. While machine learning offers an alternative, the lack of reliable data does not make it easy to apply to COVID-19 either. In this paper we present an approach that combines the strengths of traditional AI expert systems and novel deep learning models. In doing so we can leverage prior knowledge as well as any amount of existing data to quickly derive models that best adapt to the current state of the world and latest scientific knowledge. We use the approach to train a COVID-19 aware differential diagnosis model that can be used for medical decision support both for doctors or patients. We show that our approach is able to accurately model new incoming data about COVID-19 while still preserving accuracy on conditions that had been modeled in the past. While our approach shows evident and clear advantages for an extreme situation like the one we are currently facing, we also show that its flexibility generalizes beyond this concrete, but very important, example. △ Less

Submitted 30 November, 2020; v1 submitted 7 August, 2020; originally announced August 2020.

Comments: Accepted at the Machine Learning for Health (ML4H) at NeurIPS 2020 - Extended Abstract

Open Set Medical Diagnosis

Authors: Viraj Prabhu, Anitha Kannan, Geoffrey J. Tso, Namit Katariya, Manish Chablani, David Sontag, Xavier Amatriain

Abstract: Machine-learned diagnosis models have shown promise as medical aides but are trained under a closed-set assumption, i.e. that models will only encounter conditions on which they have been trained. However, it is practically infeasible to obtain sufficient training data for every human condition, and once deployed such models will invariably face previously unseen conditions. We frame machine-learn… ▽ More Machine-learned diagnosis models have shown promise as medical aides but are trained under a closed-set assumption, i.e. that models will only encounter conditions on which they have been trained. However, it is practically infeasible to obtain sufficient training data for every human condition, and once deployed such models will invariably face previously unseen conditions. We frame machine-learned diagnosis as an open-set learning problem, and study how state-of-the-art approaches compare. Further, we extend our study to a setting where training data is distributed across several healthcare sites that do not allow data pooling, and experiment with different strategies of building open-set diagnostic ensembles. Across both settings, we observe consistent gains from explicitly modeling unseen conditions, but find the optimal training strategy to vary across settings. △ Less

Submitted 7 October, 2019; originally announced October 2019.

Comments: Abbreviated version to appear at Machine Learning for Healthcare (ML4H) Workshop at NeurIPS 2019

Bottleneck Supervised U-Net for Pixel-wise Liver and Tumor Segmentation

Authors: Song Li, Geoffrey Kwok Fai Tso

Abstract: In this paper, we propose a bottleneck supervised (BS) U-Net model for liver and tumor segmentation. Our main contributions are: first, we propose a variation of the original U-Net that incorporates dense modules, inception modules and dilated convolution in the encoding path; second, we propose a bottleneck supervised (BS) U-Net that contains an encoding U-Net and a segmentation U-Net. To train t… ▽ More In this paper, we propose a bottleneck supervised (BS) U-Net model for liver and tumor segmentation. Our main contributions are: first, we propose a variation of the original U-Net that incorporates dense modules, inception modules and dilated convolution in the encoding path; second, we propose a bottleneck supervised (BS) U-Net that contains an encoding U-Net and a segmentation U-Net. To train the BS U-Net, the encoding U-Net is first trained to get encodings of the label maps that contain the anatomical information (shape and location). Subsequently, this information is used to guide the training of the segmentation U-Net so as to reserve the anatomical features of the target objects. More specifically, the loss function for segmentation U-Net is set to be the weighted average of the dice loss and the MSE loss between the encodings and the bottleneck feature vectors. The model is applied to a public liver and tumor CT scan dataset. Experimental results show that besides achieving excellent overall segmentation performance, BS U-Net also works great in controlling shape distortion, reducing false positive and false negative cases. △ Less

Submitted 12 March, 2019; v1 submitted 16 October, 2018; originally announced October 2018.

Comments: 21 pages, 9 figures, 3 tables

Learning from the experts: From expert systems to machine-learned diagnosis models

Authors: Murali Ravuri, Anitha Kannan, Geoffrey J. Tso, Xavier Amatriain

Abstract: Expert diagnostic support systems have been extensively studied. The practical applications of these systems in real-world scenarios have been somewhat limited due to well-understood shortcomings, such as lack of extensibility. More recently, machine-learned models for medical diagnosis have gained momentum, since they can learn and generalize patterns found in very large datasets like electronic… ▽ More Expert diagnostic support systems have been extensively studied. The practical applications of these systems in real-world scenarios have been somewhat limited due to well-understood shortcomings, such as lack of extensibility. More recently, machine-learned models for medical diagnosis have gained momentum, since they can learn and generalize patterns found in very large datasets like electronic health records. These models also have shortcomings - in particular, there is no easy way to incorporate prior knowledge from existing literature or experts. In this paper, we present a method to merge both approaches by using expert systems as generative models that create simulated data on which models can be learned. We demonstrate that such a learned model not only preserves the original properties of the expert systems but also addresses some of their limitations. Furthermore, we show how this approach can also be used as the starting point to combine expert knowledge with knowledge extracted from other data sources, such as electronic health records. △ Less

Submitted 14 August, 2018; v1 submitted 21 April, 2018; originally announced April 2018.