-
ChipNeMo: Domain-Adapted LLMs for Chip Design
Authors:
Mingjie Liu,
Teodor-Dumitru Ene,
Robert Kirby,
Chris Cheng,
Nathaniel Pinckney,
Rongjian Liang,
Jonah Alben,
Himyanshu Anand,
Sanmitra Banerjee,
Ismet Bayraktaroglu,
Bonita Bhaskaran,
Bryan Catanzaro,
Arjun Chaudhuri,
Sharon Clay,
Bill Dally,
Laura Dang,
Parikshit Deshpande,
Siddhanth Dhodhi,
Sameer Halepete,
Eric Hill,
Jiashang Hu,
Sumit Jain,
Ankit Jindal,
Brucek Khailany,
George Kokai
, et al. (17 additional authors not shown)
Abstract:
ChipNeMo aims to explore the applications of large language models (LLMs) for industrial chip design. Instead of directly deploying off-the-shelf commercial or open-source LLMs, we instead adopt the following domain adaptation techniques: domain-adaptive tokenization, domain-adaptive continued pretraining, model alignment with domain-specific instructions, and domain-adapted retrieval models. We e…
▽ More
ChipNeMo aims to explore the applications of large language models (LLMs) for industrial chip design. Instead of directly deploying off-the-shelf commercial or open-source LLMs, we instead adopt the following domain adaptation techniques: domain-adaptive tokenization, domain-adaptive continued pretraining, model alignment with domain-specific instructions, and domain-adapted retrieval models. We evaluate these methods on three selected LLM applications for chip design: an engineering assistant chatbot, EDA script generation, and bug summarization and analysis. Our evaluations demonstrate that domain-adaptive pretraining of language models, can lead to superior performance in domain related downstream tasks compared to their base LLaMA2 counterparts, without degradations in generic capabilities. In particular, our largest model, ChipNeMo-70B, outperforms the highly capable GPT-4 on two of our use cases, namely engineering assistant chatbot and EDA scripts generation, while exhibiting competitive performance on bug summarization and analysis. These results underscore the potential of domain-specific customization for enhancing the effectiveness of large language models in specialized applications.
△ Less
Submitted 4 April, 2024; v1 submitted 31 October, 2023;
originally announced November 2023.
-
Gradient-based Maximally Interfered Retrieval for Domain Incremental 3D Object Detection
Authors:
Barza Nisar,
Hruday Vishal Kanna Anand,
Steven L. Waslander
Abstract:
Accurate 3D object detection in all weather conditions remains a key challenge to enable the widespread deployment of autonomous vehicles, as most work to date has been performed on clear weather data. In order to generalize to adverse weather conditions, supervised methods perform best if trained from scratch on all weather data instead of finetuning a model pretrained on clear weather data. Trai…
▽ More
Accurate 3D object detection in all weather conditions remains a key challenge to enable the widespread deployment of autonomous vehicles, as most work to date has been performed on clear weather data. In order to generalize to adverse weather conditions, supervised methods perform best if trained from scratch on all weather data instead of finetuning a model pretrained on clear weather data. Training from scratch on all data will eventually become computationally infeasible and expensive as datasets continue to grow and encompass the full extent of possible weather conditions. On the other hand, naive finetuning on data from a different weather domain can result in catastrophic forgetting of the previously learned domain. Inspired by the success of replay-based continual learning methods, we propose Gradient-based Maximally Interfered Retrieval (GMIR), a gradient based sampling strategy for replay. During finetuning, GMIR periodically retrieves samples from the previous domain dataset whose gradient vectors show maximal interference with the gradient vector of the current update. Our 3D object detection experiments on the SeeingThroughFog (STF) dataset show that GMIR not only overcomes forgetting but also offers competitive performance compared to scratch training on all data with a 46.25% reduction in total training time.
△ Less
Submitted 3 May, 2023; v1 submitted 27 April, 2023;
originally announced April 2023.
-
LIP: Lightweight Intelligent Preprocessor for meaningful text-to-speech
Authors:
Harshvardhan Anand,
Nansi Begam,
Richa Verma,
Sourav Ghosh,
Harichandana B. S. S,
Sumit Kumar
Abstract:
Existing Text-to-Speech (TTS) systems need to read messages from the email which may have Personal Identifiable Information (PII) to text messages that can have a streak of emojis and punctuation. 92% of the world's online population use emoji with more than 10 billion emojis sent everyday. Lack of preprocessor leads to messages being read as-is including punctuation and infographics like emoticon…
▽ More
Existing Text-to-Speech (TTS) systems need to read messages from the email which may have Personal Identifiable Information (PII) to text messages that can have a streak of emojis and punctuation. 92% of the world's online population use emoji with more than 10 billion emojis sent everyday. Lack of preprocessor leads to messages being read as-is including punctuation and infographics like emoticons. This problem worsens if there is a continuous sequence of punctuation/emojis that are quite common in real-world communications like messaging, Social Networking Site (SNS) interactions, etc. In this work, we aim to introduce a lightweight intelligent preprocessor (LIP) that can enhance the readability of a message before being passed downstream to existing TTS systems. We propose multiple sub-modules including: expanding contraction, censoring swear words, and masking of PII, as part of our preprocessor to enhance the readability of text. With a memory footprint of only 3.55 MB and inference time of 4 ms for up to 50-character text, our solution is suitable for real-time deployment. This work being the first of its kind, we try to benchmark with an open independent survey, the result of which shows 76.5% preference towards LIP enabled TTS engine as compared to standard TTS.
△ Less
Submitted 11 July, 2022;
originally announced July 2022.
-
Application of Machine Learning to Sleep Stage Classification
Authors:
Andrew Smith,
Hardik Anand,
Snezana Milosavljevic,
Katherine M. Rentschler,
Ana Pocivavsek,
Homayoun Valafar
Abstract:
Sleep studies are imperative to recapitulate phenotypes associated with sleep loss and uncover mechanisms contributing to psychopathology. Most often, investigators manually classify the polysomnography into vigilance states, which is time-consuming, requires extensive training, and is prone to inter-scorer variability. While many works have successfully developed automated vigilance state classif…
▽ More
Sleep studies are imperative to recapitulate phenotypes associated with sleep loss and uncover mechanisms contributing to psychopathology. Most often, investigators manually classify the polysomnography into vigilance states, which is time-consuming, requires extensive training, and is prone to inter-scorer variability. While many works have successfully developed automated vigilance state classifiers based on multiple EEG channels, we aim to produce an automated and open-access classifier that can reliably predict vigilance state based on a single cortical electroencephalogram (EEG) from rodents to minimize the disadvantages that accompany tethering small animals via wires to computer programs. Approximately 427 hours of continuously monitored EEG, electromyogram (EMG), and activity were labeled by a domain expert out of 571 hours of total data. Here we evaluate the performance of various machine learning techniques on classifying 10-second epochs into one of three discrete classes: paradoxical, slow-wave, or wake. Our investigations include Decision Trees, Random Forests, Naive Bayes Classifiers, Logistic Regression Classifiers, and Artificial Neural Networks. These methodologies have achieved accuracies ranging from approximately 74% to approximately 96%. Most notably, the Random Forest and the ANN achieved remarkable accuracies of 95.78% and 93.31%, respectively. Here we have shown the potential of various machine learning classifiers to automatically, accurately, and reliably classify vigilance states based on a single EEG reading and a single EMG reading.
△ Less
Submitted 22 May, 2022; v1 submitted 4 November, 2021;
originally announced November 2021.
-
Robotics During a Pandemic: The 2020 NSF CPS Virtual Challenge -- SoilScope, Mars Edition
Authors:
Darwin Mick,
K. Srikar Siddarth,
Swastik Nandan,
Harish Anand,
Stephen A. Rees,
Jnaneshwar Das
Abstract:
Remote sample recovery is a rapidly evolving application of Small Unmanned Aircraft Systems (sUAS) for planetary sciences and space exploration. Development of cyber-physical systems (CPS) for autonomous deployment and recovery of sensor probes for sample caching is already in progress with NASA's MARS 2020 mission. To challenge student teams to develop autonomy for sample recovery settings, the 2…
▽ More
Remote sample recovery is a rapidly evolving application of Small Unmanned Aircraft Systems (sUAS) for planetary sciences and space exploration. Development of cyber-physical systems (CPS) for autonomous deployment and recovery of sensor probes for sample caching is already in progress with NASA's MARS 2020 mission. To challenge student teams to develop autonomy for sample recovery settings, the 2020 NSF CPS Challenge was positioned around the launch of the MARS 2020 rover and sUAS duo. This paper discusses perception and trajectory planning for sample recovery by sUAS in a simulation environment. Out of a total of five teams that participated, the results of the top two teams have been discussed. The OpenUAV cloud simulation framework deployed on the Cyber-Physical Systems Virtual Organization (CPS-VO) allowed the teams to work remotely over a month during the COVID-19 pandemic to develop and simulate autonomous exploration algorithms. Remote simulation enabled teams across the globe to collaborate in experiments. The two teams approached the task of probe search, probe recovery, and landing on a moving target differently. This paper is a summary of teams' insights and lessons learned, as they chose from a wide range of perception sensors and algorithms.
△ Less
Submitted 15 March, 2021;
originally announced March 2021.
-
OpenUAV Cloud Testbed: a Collaborative Design Studio for Field Robotics
Authors:
Harish Anand,
Stephen A. Rees,
Zhiang Chen,
Ashwin Jose,
Sarah Bearman,
Prasad Antervedi,
Jnaneshwar Das
Abstract:
Simulations play a crucial role in robotics research and education. This paper presents the OpenUAV testbed, an open-source, easy-to-use, web-based, and reproducible software system that enables students and researchers to run robotic simulations on the cloud. We have built upon our previous work and have addressed some of the educational and research challenges associated with the prior work. The…
▽ More
Simulations play a crucial role in robotics research and education. This paper presents the OpenUAV testbed, an open-source, easy-to-use, web-based, and reproducible software system that enables students and researchers to run robotic simulations on the cloud. We have built upon our previous work and have addressed some of the educational and research challenges associated with the prior work. The critical contributions of the paper to the robotics and automation community are threefold: First, OpenUAV saves students and researchers from tedious and complicated software setups by providing web-browser-based Linux desktop sessions with standard robotics software like Gazebo, ROS, and flight autonomy stack. Second, a method for saving an individual's research work with its dependencies for the work's future reproducibility. Third, the platform provides a mechanism to support photorealistic robotics simulations by combining Unity game engine-based camera rendering and Gazebo physics. The paper addresses a research need for photorealistic simulations and describes a methodology for creating a photorealistic aquatic simulation. We also present the various academic and research use-cases of this platform to improve robotics education and research, especially during times like the COVID-19 pandemic, when virtual collaboration is necessary.
△ Less
Submitted 6 May, 2021; v1 submitted 1 October, 2019;
originally announced October 2019.
-
Geomorphological Analysis Using Unpiloted Aircraft Systems, Structure from Motion, and Deep Learning
Authors:
Zhiang Chen,
Tyler R. Scott,
Sarah Bearman,
Harish Anand,
Devin Keating,
Chelsea Scott,
J Ramon Arrowsmith,
Jnaneshwar Das
Abstract:
We present a pipeline for geomorphological analysis that uses structure from motion (SfM) and deep learning on close-range aerial imagery to estimate spatial distributions of rock traits (size, roundness, and orientation) along a tectonic fault scarp. The properties of the rocks on the fault scarp derive from the combination of initial volcanic fracturing and subsequent tectonic and geomorphic fra…
▽ More
We present a pipeline for geomorphological analysis that uses structure from motion (SfM) and deep learning on close-range aerial imagery to estimate spatial distributions of rock traits (size, roundness, and orientation) along a tectonic fault scarp. The properties of the rocks on the fault scarp derive from the combination of initial volcanic fracturing and subsequent tectonic and geomorphic fracturing, and our pipeline allows scientists to leverage UAS-based imagery to gain a better understanding of such surface processes. We start by using SfM on aerial imagery to produce georeferenced orthomosaics and digital elevation models (DEM). A human expert then annotates rocks on a set of image tiles sampled from the orthomosaics, and these annotations are used to train a deep neural network to detect and segment individual rocks in the entire site. The extracted semantic information (rock masks) on large volumes of unlabeled, high-resolution SfM products allows subsequent structural analysis and shape descriptors to estimate rock size, roundness, and orientation. We present results of two experiments conducted along a fault scarp in the Volcanic Tablelands near Bishop, California. We conducted the first, proof-of-concept experiment with a DJI Phantom 4 Pro equipped with an RGB camera and inspected if elevation information assisted instance segmentation from RGB channels. Rock-trait histograms along and across the fault scarp were obtained with the neural network inference. In the second experiment, we deployed a hexrotor and a multispectral camera to produce a DEM and five spectral orthomosaics in red, green, blue, red edge, and near infrared. We focused on examining the effectiveness of different combinations of input channels in instance segmentation.
△ Less
Submitted 17 February, 2021; v1 submitted 27 September, 2019;
originally announced September 2019.
