-
GRAD-SUM: Leveraging Gradient Summarization for Optimal Prompt Engineering
Authors:
Derek Austin,
Elliott Chartock
Abstract:
Prompt engineering for large language models (LLMs) is often a manual time-intensive process that involves generating, evaluating, and refining prompts iteratively to ensure high-quality outputs. While there has been work on automating prompt engineering, the solutions generally are either tuned to specific tasks with given answers or are quite costly. We introduce GRAD-SUM, a scalable and flexibl…
▽ More
Prompt engineering for large language models (LLMs) is often a manual time-intensive process that involves generating, evaluating, and refining prompts iteratively to ensure high-quality outputs. While there has been work on automating prompt engineering, the solutions generally are either tuned to specific tasks with given answers or are quite costly. We introduce GRAD-SUM, a scalable and flexible method for automatic prompt engineering that builds on gradient-based optimization techniques. Our approach incorporates user-defined task descriptions and evaluation criteria, and features a novel gradient summarization module to generalize feedback effectively. Our results demonstrate that GRAD-SUM consistently outperforms existing methods across various benchmarks, highlighting its versatility and effectiveness in automatic prompt optimization.
△ Less
Submitted 12 July, 2024;
originally announced July 2024.
-
Bayesian Optimization with LLM-Based Acquisition Functions for Natural Language Preference Elicitation
Authors:
David Eric Austin,
Anton Korikov,
Armin Toroghi,
Scott Sanner
Abstract:
Designing preference elicitation (PE) methodologies that can quickly ascertain a user's top item preferences in a cold-start setting is a key challenge for building effective and personalized conversational recommendation (ConvRec) systems. While large language models (LLMs) enable fully natural language (NL) PE dialogues, we hypothesize that monolithic LLM NL-PE approaches lack the multi-turn, de…
▽ More
Designing preference elicitation (PE) methodologies that can quickly ascertain a user's top item preferences in a cold-start setting is a key challenge for building effective and personalized conversational recommendation (ConvRec) systems. While large language models (LLMs) enable fully natural language (NL) PE dialogues, we hypothesize that monolithic LLM NL-PE approaches lack the multi-turn, decision-theoretic reasoning required to effectively balance the exploration and exploitation of user preferences towards an arbitrary item set. In contrast, traditional Bayesian optimization PE methods define theoretically optimal PE strategies, but cannot generate arbitrary NL queries or reason over content in NL item descriptions -- requiring users to express preferences via ratings or comparisons of unfamiliar items. To overcome the limitations of both approaches, we formulate NL-PE in a Bayesian Optimization (BO) framework that seeks to actively elicit NL feedback to identify the best recommendation. Key challenges in generalizing BO to deal with natural language feedback include determining: (a) how to leverage LLMs to model the likelihood of NL preference feedback as a function of item utilities, and (b) how to design an acquisition function for NL BO that can elicit preferences in the infinite space of language. We demonstrate our framework in a novel NL-PE algorithm, PEBOL, which uses: 1) Natural Language Inference (NLI) between user preference utterances and NL item descriptions to maintain Bayesian preference beliefs, and 2) BO strategies such as Thompson Sampling (TS) and Upper Confidence Bound (UCB) to steer LLM query generation. We numerically evaluate our methods in controlled simulations, finding that after 10 turns of dialogue, PEBOL can achieve an MRR@10 of up to 0.27 compared to the best monolithic LLM baseline's MRR@10 of 0.17, despite relying on earlier and smaller LLMs.
△ Less
Submitted 19 August, 2024; v1 submitted 1 May, 2024;
originally announced May 2024.
-
MEDBind: Unifying Language and Multimodal Medical Data Embeddings
Authors:
Yuan Gao,
Sangwook Kim,
David E Austin,
Chris McIntosh
Abstract:
Medical vision-language pretraining models (VLPM) have achieved remarkable progress in fusing chest X-rays (CXR) with clinical texts, introducing image-text data binding approaches that enable zero-shot learning and downstream clinical tasks. However, the current landscape lacks the holistic integration of additional medical modalities, such as electrocardiograms (ECG). We present MEDBind (Medical…
▽ More
Medical vision-language pretraining models (VLPM) have achieved remarkable progress in fusing chest X-rays (CXR) with clinical texts, introducing image-text data binding approaches that enable zero-shot learning and downstream clinical tasks. However, the current landscape lacks the holistic integration of additional medical modalities, such as electrocardiograms (ECG). We present MEDBind (Medical Electronic patient recorD), which learns joint embeddings across CXR, ECG, and medical text. Using text data as the central anchor, MEDBind features tri-modality binding, delivering competitive performance in top-K retrieval, zero-shot, and few-shot benchmarks against established VLPM, and the ability for CXR-to-ECG zero-shot classification and retrieval. This seamless integration is achieved through combination of contrastive loss on modality-text pairs with our proposed contrastive loss function, Edge-Modality Contrastive Loss, fostering a cohesive embedding space for CXR, ECG, and text. Finally, we demonstrate that MEDBind can improve downstream tasks by directly integrating CXR and ECG embeddings into a large-language model for multimodal prompt tuning.
△ Less
Submitted 20 March, 2024; v1 submitted 19 March, 2024;
originally announced March 2024.
-
FB-OCC: 3D Occupancy Prediction based on Forward-Backward View Transformation
Authors:
Zhiqi Li,
Zhiding Yu,
David Austin,
Mingsheng Fang,
Shiyi Lan,
Jan Kautz,
Jose M. Alvarez
Abstract:
This technical report summarizes the winning solution for the 3D Occupancy Prediction Challenge, which is held in conjunction with the CVPR 2023 Workshop on End-to-End Autonomous Driving and CVPR 23 Workshop on Vision-Centric Autonomous Driving Workshop. Our proposed solution FB-OCC builds upon FB-BEV, a cutting-edge camera-based bird's-eye view perception design using forward-backward projection.…
▽ More
This technical report summarizes the winning solution for the 3D Occupancy Prediction Challenge, which is held in conjunction with the CVPR 2023 Workshop on End-to-End Autonomous Driving and CVPR 23 Workshop on Vision-Centric Autonomous Driving Workshop. Our proposed solution FB-OCC builds upon FB-BEV, a cutting-edge camera-based bird's-eye view perception design using forward-backward projection. On top of FB-BEV, we further study novel designs and optimization tailored to the 3D occupancy prediction task, including joint depth-semantic pre-training, joint voxel-BEV representation, model scaling up, and effective post-processing strategies. These designs and optimization result in a state-of-the-art mIoU score of 54.19% on the nuScenes dataset, ranking the 1st place in the challenge track. Code and models will be released at: https://github.com/NVlabs/FB-BEV.
△ Less
Submitted 4 July, 2023;
originally announced July 2023.
-
Surgical tool classification and localization: results and methods from the MICCAI 2022 SurgToolLoc challenge
Authors:
Aneeq Zia,
Kiran Bhattacharyya,
Xi Liu,
Max Berniker,
Ziheng Wang,
Rogerio Nespolo,
Satoshi Kondo,
Satoshi Kasai,
Kousuke Hirasawa,
Bo Liu,
David Austin,
Yiheng Wang,
Michal Futrega,
Jean-Francois Puget,
Zhenqiang Li,
Yoichi Sato,
Ryo Fujii,
Ryo Hachiuma,
Mana Masuda,
Hideo Saito,
An Wang,
Mengya Xu,
Mobarakol Islam,
Long Bai,
Winnie Pang
, et al. (46 additional authors not shown)
Abstract:
The ability to automatically detect and track surgical instruments in endoscopic videos can enable transformational interventions. Assessing surgical performance and efficiency, identifying skilled tool use and choreography, and planning operational and logistical aspects of OR resources are just a few of the applications that could benefit. Unfortunately, obtaining the annotations needed to train…
▽ More
The ability to automatically detect and track surgical instruments in endoscopic videos can enable transformational interventions. Assessing surgical performance and efficiency, identifying skilled tool use and choreography, and planning operational and logistical aspects of OR resources are just a few of the applications that could benefit. Unfortunately, obtaining the annotations needed to train machine learning models to identify and localize surgical tools is a difficult task. Annotating bounding boxes frame-by-frame is tedious and time-consuming, yet large amounts of data with a wide variety of surgical tools and surgeries must be captured for robust training. Moreover, ongoing annotator training is needed to stay up to date with surgical instrument innovation. In robotic-assisted surgery, however, potentially informative data like timestamps of instrument installation and removal can be programmatically harvested. The ability to rely on tool installation data alone would significantly reduce the workload to train robust tool-tracking models. With this motivation in mind we invited the surgical data science community to participate in the challenge, SurgToolLoc 2022. The goal was to leverage tool presence data as weak labels for machine learning models trained to detect tools and localize them in video frames with bounding boxes. We present the results of this challenge along with many of the team's efforts. We conclude by discussing these results in the broader context of machine learning and surgical data science. The training data used for this challenge consisting of 24,695 video clips with tool presence labels is also being released publicly and can be accessed at https://console.cloud.google.com/storage/browser/isi-surgtoolloc-2022.
△ Less
Submitted 31 May, 2023; v1 submitted 11 May, 2023;
originally announced May 2023.
-
From Human Days to Machine Seconds: Automatically Answering and Generating Machine Learning Final Exams
Authors:
Iddo Drori,
Sarah J. Zhang,
Reece Shuttleworth,
Sarah Zhang,
Keith Tyser,
Zad Chin,
Pedro Lantigua,
Saisamrit Surbehera,
Gregory Hunter,
Derek Austin,
Leonard Tang,
Yann Hicke,
Sage Simhon,
Sathwik Karnik,
Darnell Granberry,
Madeleine Udell
Abstract:
A final exam in machine learning at a top institution such as MIT, Harvard, or Cornell typically takes faculty days to write, and students hours to solve. We demonstrate that large language models pass machine learning finals at a human level, on finals available online after the models were trained, and automatically generate new human-quality final exam questions in seconds. Previous work has de…
▽ More
A final exam in machine learning at a top institution such as MIT, Harvard, or Cornell typically takes faculty days to write, and students hours to solve. We demonstrate that large language models pass machine learning finals at a human level, on finals available online after the models were trained, and automatically generate new human-quality final exam questions in seconds. Previous work has developed program synthesis and few-shot learning methods to solve university-level problem set questions in mathematics and STEM courses. In this work, we develop and compare methods that solve final exams, which differ from problem sets in several ways: the questions are longer, have multiple parts, are more complicated, and span a broader set of topics. We curate a dataset and benchmark of questions from machine learning final exams available online and code for answering these questions and generating new questions. We show how to generate new questions from other questions and course notes. For reproducibility and future research on this final exam benchmark, we use automatic checkers for multiple-choice, numeric, and questions with expression answers. We perform ablation studies comparing zero-shot learning with few-shot learning and chain-of-thought prompting using GPT-3, OPT, Codex, and ChatGPT across machine learning topics and find that few-shot learning methods perform best. We highlight the transformative potential of language models to streamline the writing and solution of large-scale assessments, significantly reducing the workload from human days to mere machine seconds. Our results suggest that rather than banning large language models such as ChatGPT in class, instructors should teach students to harness them by asking students meta-questions about correctness, completeness, and originality of the responses generated, encouraging critical thinking in academic studies.
△ Less
Submitted 28 June, 2023; v1 submitted 11 June, 2022;
originally announced June 2022.
-
Connecting the Dots in Nutritional Rehabilitation: A Qualitative Study on ICT and Community Based Care
Authors:
Deepa Austin,
Amit Prakash
Abstract:
'Fragmentation in care' continuum is often considered as a shortcoming of Health system whereas, 'Integration of care' is widely acclaimed as a viable solution to fragmentation. In last two decades, Information and communication technologies (ICTs), by virtue of their ability to integrate information for action, has been extensively used in addressing many public health problems like malnutrition.…
▽ More
'Fragmentation in care' continuum is often considered as a shortcoming of Health system whereas, 'Integration of care' is widely acclaimed as a viable solution to fragmentation. In last two decades, Information and communication technologies (ICTs), by virtue of their ability to integrate information for action, has been extensively used in addressing many public health problems like malnutrition. Tackling the public health challenge of malnutrition demands attention to interconnectedness and interactions between multiple systems. In this paper, using a case study of an ICT application used by community workers for malnutrition management in Karnataka, we argue that lack of recognition of interconnectedness and interactions among stakeholders and context can pose a challenge to integration of care. ICTs can be key enablers to overcome fragmentation, provided it recognizes the inherent complexities of malnutrition and its management. We argue that for an effective ICT enabled integration of Severe Acute Malnutrition (SAM) management, a thorough understanding of perspectives of multiple stakeholders together with rich picture of the contextual dynamics should not be ignored at design and implementation phase.
△ Less
Submitted 22 August, 2021;
originally announced August 2021.
-
Learning to Localize Temporal Events in Large-scale Video Data
Authors:
Mikel Bober-Irizar,
Miha Skalic,
David Austin
Abstract:
We address temporal localization of events in large-scale video data, in the context of the Youtube-8M Segments dataset. This emerging field within video recognition can enable applications to identify the precise time a specified event occurs in a video, which has broad implications for video search. To address this we present two separate approaches: (1) a gradient boosted decision tree model on…
▽ More
We address temporal localization of events in large-scale video data, in the context of the Youtube-8M Segments dataset. This emerging field within video recognition can enable applications to identify the precise time a specified event occurs in a video, which has broad implications for video search. To address this we present two separate approaches: (1) a gradient boosted decision tree model on a crafted dataset and (2) a combination of deep learning models based on frame-level data, video-level data, and a localization model. The combinations of these two approaches achieved 5th place in the 3rd Youtube-8M video recognition challenge.
△ Less
Submitted 25 October, 2019;
originally announced October 2019.
-
Computer-aided molecular design: An introduction and review of tools, applications, and solution techniques
Authors:
Nick D. Austin,
Nikolaos V. Sahinidis,
Daniel W. Trahan
Abstract:
This article provides an introduction to and review of the field of computer-aided molecular design (CAMD). It is intended to be approachable for the absolute beginner as well as useful to the seasoned CAMD practitioner. We begin by discussing various quantitative structure-property relationships (QSPRs) which have been demonstrated to work well with CAMD problems. The methods discussed in this ar…
▽ More
This article provides an introduction to and review of the field of computer-aided molecular design (CAMD). It is intended to be approachable for the absolute beginner as well as useful to the seasoned CAMD practitioner. We begin by discussing various quantitative structure-property relationships (QSPRs) which have been demonstrated to work well with CAMD problems. The methods discussed in this article are (1) group contribution methods, (2) topological indices, and (3) signature descriptors. Next, we present general optimization formulations for various forms of the CAMD problem. Common design constraints are discussed and structural feasibility constraints are provided for the three types of QSPRs addressed. We then detail useful techniques for approaching CAMD optimization problems, including decomposition methods, heuristic approaches, and mathematical programming strategies. Finally, we discuss many applications that have been addressed using CAMD.
△ Less
Submitted 14 January, 2017;
originally announced January 2017.
-
Gait Velocity Estimation using time interleaved between Consecutive Passive IR Sensor Activations
Authors:
Rajib Rana,
Daniel Austin,
Peter G. Jacobs,
Mohanraj Karunanithi,
Jeffrey Kaye
Abstract:
Gait velocity has been consistently shown to be an important indicator and predictor of health status, especially in older adults. It is often assessed clinically, but the assessments occur infrequently and do not allow optimal detection of key health changes when they occur. In this paper, we show that the time gap between activations of a pair of Passive Infrared (PIR) motion sensors installed i…
▽ More
Gait velocity has been consistently shown to be an important indicator and predictor of health status, especially in older adults. It is often assessed clinically, but the assessments occur infrequently and do not allow optimal detection of key health changes when they occur. In this paper, we show that the time gap between activations of a pair of Passive Infrared (PIR) motion sensors installed in the consecutively visited room pair carry rich latent information about a person's gait velocity. We name this time gap transition time and show that despite a six second refractory period of the PIR sensors, transition time can be used to obtain an accurate representation of gait velocity.
Using a Support Vector Regression (SVR) approach to model the relationship between transition time and gait velocity, we show that gait velocity can be estimated with an average error less than 2.5 cm/sec. This is demonstrated with data collected over a 5 year period from 74 older adults monitored in their own homes.
This method is simple and cost effective and has advantages over competing approaches such as: obtaining 20 to 100x more gait velocity measurements per day and offering the fusion of location-specific information with time stamped gait estimates. These advantages allow stable estimates of gait parameters (maximum or average speed, variability) at shorter time scales than current approaches. This also provides a pervasive in-home method for context-aware gait velocity sensing that allows for monitoring of gait trajectories in space and time.
△ Less
Submitted 30 November, 2015; v1 submitted 17 October, 2013;
originally announced October 2013.
