-
Spectra: A Comprehensive Study of Ternary, Quantized, and FP16 Language Models
Authors:
Ayush Kaushal,
Tejas Pandey,
Tejas Vaidhya,
Aaryan Bhagat,
Irina Rish
Abstract:
Post-training quantization is the leading method for addressing memory-related bottlenecks in LLM inference, but unfortunately, it suffers from significant performance degradation below 4-bit precision. An alternative approach involves training compressed models directly at a low bitwidth (e.g., binary or ternary models). However, the performance, training dynamics, and scaling trends of such mode…
▽ More
Post-training quantization is the leading method for addressing memory-related bottlenecks in LLM inference, but unfortunately, it suffers from significant performance degradation below 4-bit precision. An alternative approach involves training compressed models directly at a low bitwidth (e.g., binary or ternary models). However, the performance, training dynamics, and scaling trends of such models are not yet well understood. To address this issue, we train and openly release the Spectra LLM suite consisting of 54 language models ranging from 99M to 3.9B parameters, trained on 300B tokens. Spectra includes FloatLMs, post-training quantized QuantLMs (3, 4, 6, and 8 bits), and ternary LLMs (TriLMs) - our improved architecture for ternary language modeling, which significantly outperforms previously proposed ternary models of a given size (in bits), matching half-precision models at scale. For example, TriLM 3.9B is (bit-wise) smaller than the half-precision FloatLM 830M, but matches half-precision FloatLM 3.9B in commonsense reasoning and knowledge benchmarks. However, TriLM 3.9B is also as toxic and stereotyping as FloatLM 3.9B, a model six times larger in size. Additionally, TriLM 3.9B lags behind FloatLM in perplexity on validation splits and web-based corpora but performs better on less noisy datasets like Lambada and PennTreeBank.
To enhance understanding of low-bitwidth models, we are releasing 500+ intermediate checkpoints of the Spectra suite at \href{https://github.com/NolanoOrg/SpectraSuite}{https://github.com/NolanoOrg/SpectraSuite}.
△ Less
Submitted 17 July, 2024;
originally announced July 2024.
-
IITP-VDLand: A Comprehensive Dataset on Decentraland Parcels
Authors:
Ankit K. Bhagat,
Dipika Jha,
Raju Halder,
Rajendra N. Paramanik,
Chandra M. Kumar
Abstract:
This paper presents IITP-VDLand, a comprehensive dataset of Decentraland parcels sourced from diverse platforms. Unlike existing datasets which have limited attributes and records, IITP-VDLand offers a rich array of attributes, encompassing parcel characteristics, trading history, past activities, transactions, and social media interactions. Alongside, we introduce a key attribute in the dataset,…
▽ More
This paper presents IITP-VDLand, a comprehensive dataset of Decentraland parcels sourced from diverse platforms. Unlike existing datasets which have limited attributes and records, IITP-VDLand offers a rich array of attributes, encompassing parcel characteristics, trading history, past activities, transactions, and social media interactions. Alongside, we introduce a key attribute in the dataset, namely Rarity score, which measures the uniqueness of each parcel within the virtual world. Addressing the significant challenge posed by the dispersed nature of this data across various sources, we employ a systematic approach, utilizing both available APIs and custom scripts, to gather it. Subsequently, we meticulously curate and organize the information into four distinct segments: (1) Characteristics Data-Fragment, (2) OpenSea Trading History Data-Fragment, (3) Ethereum Activity Transactions Data-Fragment, and (4) Social Media Data-Fragment. We envisage that this dataset would serve as a robust resource for training machine- and deep-learning models specifically designed to address real-world challenges within the domain of Decentraland parcels. The performance benchmarking of more than 20 state-of-the-art price prediction models on our dataset yields promising results, achieving a maximum R2 score of 0.8251 and an accuracy of 74.23% in case of Extra Trees Regressor and Classifier. The key findings reveal that the ensemble models performs better than both deep learning and linear models for our dataset. We observe a significant impact of coordinates, geographical proximity, rarity score, and few other economic indicators on the prediction of parcel prices.
△ Less
Submitted 11 April, 2024;
originally announced April 2024.
-
Emotional Intelligence Through Artificial Intelligence : NLP and Deep Learning in the Analysis of Healthcare Texts
Authors:
Prashant Kumar Nag,
Amit Bhagat,
R. Vishnu Priya,
Deepak kumar Khare
Abstract:
This manuscript presents a methodical examination of the utilization of Artificial Intelligence in the assessment of emotions in texts related to healthcare, with a particular focus on the incorporation of Natural Language Processing and deep learning technologies. We scrutinize numerous research studies that employ AI to augment sentiment analysis, categorize emotions, and forecast patient outcom…
▽ More
This manuscript presents a methodical examination of the utilization of Artificial Intelligence in the assessment of emotions in texts related to healthcare, with a particular focus on the incorporation of Natural Language Processing and deep learning technologies. We scrutinize numerous research studies that employ AI to augment sentiment analysis, categorize emotions, and forecast patient outcomes based on textual information derived from clinical narratives, patient feedback on medications, and online health discussions. The review demonstrates noteworthy progress in the precision of algorithms used for sentiment classification, the prognostic capabilities of AI models for neurodegenerative diseases, and the creation of AI-powered systems that offer support in clinical decision-making. Remarkably, the utilization of AI applications has exhibited an enhancement in personalized therapy plans by integrating patient sentiment and contributing to the early identification of mental health disorders. There persist challenges, which encompass ensuring the ethical application of AI, safeguarding patient confidentiality, and addressing potential biases in algorithmic procedures. Nevertheless, the potential of AI to revolutionize healthcare practices is unmistakable, offering a future where healthcare is not only more knowledgeable and efficient but also more empathetic and centered around the needs of patients. This investigation underscores the transformative influence of AI on healthcare, delivering a comprehensive comprehension of its role in examining emotional content in healthcare texts and highlighting the trajectory towards a more compassionate approach to patient care. The findings advocate for a harmonious synergy between AI's analytical capabilities and the human aspects of healthcare.
△ Less
Submitted 14 March, 2024;
originally announced March 2024.
-
Subfield codes of $C_D$-codes over $\mathbb{F}_2[x]/\langle x^3-x \rangle$ are really nice!
Authors:
Anuj Kumar Bhagat,
Ritumoni Sarma,
Vidya Sagar
Abstract:
A non-zero $\mathbb{F}$-linear map from a finite-dimensional commutative $\mathbb{F}$-algebra to $\mathbb{F}$ is called an $\mathbb{F}$-valued trace if its kernel does not contain any non-zero ideals. In this article, we utilize an $\mathbb{F}_2$-valued trace of the $\mathbb{F}_2$-algebra $\mathcal{R}_2:=\mathbb{F}_2[x]/\langle x^3-x\rangle$ to study binary subfield code $\mathcal{C}_D^{(2)}$ of…
▽ More
A non-zero $\mathbb{F}$-linear map from a finite-dimensional commutative $\mathbb{F}$-algebra to $\mathbb{F}$ is called an $\mathbb{F}$-valued trace if its kernel does not contain any non-zero ideals. In this article, we utilize an $\mathbb{F}_2$-valued trace of the $\mathbb{F}_2$-algebra $\mathcal{R}_2:=\mathbb{F}_2[x]/\langle x^3-x\rangle$ to study binary subfield code $\mathcal{C}_D^{(2)}$ of $\mathcal{C}_D:=\{\left(x\cdot d\right)_{d\in D}: x\in \mathcal{R}_2^m\}$ for each defining set $D$ derived from a certain simplicial complex. For $m\in \mathbb{N}$ and $X\subseteq \{1, 2, \dots, m\}$, define $Δ_X:=\{v\in \mathbb{F}_2^m: \Supp(v)\subseteq X\}$ and $D:=(1+u^2)D_1+u^2D_2+(u+u^2)D_3,$ a subset of $\mathcal{R}_2^m,$ where $u=x+\langle x^3-x\rangle, D_1\in \{Δ_L, Δ_L^c\},\, D_2\in \{Δ_M, Δ_M^c\}$ and $ D_3\in \{Δ_N, Δ_N^c\}$, for $L, M, N\subseteq \{1, 2, \dots, m\}.$ The parameters and the Hamming weight distribution of the binary subfield code $\mathcal{C}_D^{(2)}$ of $\mathcal{C}_D$ are determined for each $D.$ These binary subfield codes are minimal under certain mild conditions on the cardinalities of $L, M$ and $N$. Moreover, most of these codes are distance-optimal. Consequently, we obtain a few infinite families of minimal, self-orthogonal and distance-optimal binary linear codes that are either $2$-weight or $4$-weight. It is worth mentioning that we have obtained several new distance-optimal binary linear codes.
△ Less
Submitted 16 February, 2024;
originally announced February 2024.
-
$\mathbb{F}$-valued trace of a finite-dimensional commutative $\mathbb{F}$-algebra
Authors:
Anuj Kr Bhagat,
Ritumoni Sarma
Abstract:
A non-zero $\mathbb{F}$-valued $\mathbb{F}$-linear map on a finite dimensional $\mathbb{F}$-algebra is called an $\mathbb{F}$-valued trace if its kernel does not contain any non-zero ideals. However, given an $\mathbb{F}$-algebra such a map may not always exist. We find an infinite class of finite-dimensional commutative $\mathbb{F}$-algebras which admit an $\mathbb{F}$-valued trace. In fact, in t…
▽ More
A non-zero $\mathbb{F}$-valued $\mathbb{F}$-linear map on a finite dimensional $\mathbb{F}$-algebra is called an $\mathbb{F}$-valued trace if its kernel does not contain any non-zero ideals. However, given an $\mathbb{F}$-algebra such a map may not always exist. We find an infinite class of finite-dimensional commutative $\mathbb{F}$-algebras which admit an $\mathbb{F}$-valued trace. In fact, in these cases, we explicitly construct a trace map. The existence of an $\mathbb{F}$-valued trace on a finite dimensional commutative $\mathbb{F}$-algebra induces a non-degenerate bilinear form on the $\mathbb{F}$-algebra which may be helpful both theoretically and computationally. In this article, we suggest a couple of applications of an $\mathbb{F}$-valued trace map of an $\mathbb{F}$-algebra to algebraic coding theory.
△ Less
Submitted 19 September, 2023; v1 submitted 18 September, 2023;
originally announced September 2023.
-
PBP: Path-based Trajectory Prediction for Autonomous Driving
Authors:
Sepideh Afshar,
Nachiket Deo,
Akshay Bhagat,
Titas Chakraborty,
Yunming Shao,
Balarama Raju Buddharaju,
Adwait Deshpande,
Henggang Cui
Abstract:
Trajectory prediction plays a crucial role in the autonomous driving stack by enabling autonomous vehicles to anticipate the motion of surrounding agents. Goal-based prediction models have gained traction in recent years for addressing the multimodal nature of future trajectories. Goal-based prediction models simplify multimodal prediction by first predicting 2D goal locations of agents and then p…
▽ More
Trajectory prediction plays a crucial role in the autonomous driving stack by enabling autonomous vehicles to anticipate the motion of surrounding agents. Goal-based prediction models have gained traction in recent years for addressing the multimodal nature of future trajectories. Goal-based prediction models simplify multimodal prediction by first predicting 2D goal locations of agents and then predicting trajectories conditioned on each goal. However, a single 2D goal location serves as a weak inductive bias for predicting the whole trajectory, often leading to poor map compliance, i.e., part of the trajectory going off-road or breaking traffic rules. In this paper, we improve upon goal-based prediction by proposing the Path-based prediction (PBP) approach. PBP predicts a discrete probability distribution over reference paths in the HD map using the path features and predicts trajectories in the path-relative Frenet frame. We applied the PBP trajectory decoder on top of the HiVT scene encoder and report results on the Argoverse dataset. Our experiments show that PBP achieves competitive performance on the standard trajectory prediction metrics, while significantly outperforming state-of-the-art baselines in terms of map compliance.
△ Less
Submitted 2 March, 2024; v1 submitted 7 September, 2023;
originally announced September 2023.
-
Improving Motion Forecasting for Autonomous Driving with the Cycle Consistency Loss
Authors:
Titas Chakraborty,
Akshay Bhagat,
Henggang Cui
Abstract:
Robust motion forecasting of the dynamic scene is a critical component of an autonomous vehicle. It is a challenging problem due to the heterogeneity in the scene and the inherent uncertainties in the problem. To improve the accuracy of motion forecasting, in this work, we identify a new consistency constraint in this task, that is an agent's future trajectory should be coherent with its history o…
▽ More
Robust motion forecasting of the dynamic scene is a critical component of an autonomous vehicle. It is a challenging problem due to the heterogeneity in the scene and the inherent uncertainties in the problem. To improve the accuracy of motion forecasting, in this work, we identify a new consistency constraint in this task, that is an agent's future trajectory should be coherent with its history observations and visa versa. To leverage this property, we propose a novel cycle consistency training scheme and define a novel cycle loss to encourage this consistency. In particular, we reverse the predicted future trajectory backward in time and feed it back into the prediction model to predict the history and compute the loss as an additional cycle loss term. Through our experiments on the Argoverse dataset, we demonstrate that cycle loss can improve the performance of competitive motion forecasting models.
△ Less
Submitted 31 October, 2022;
originally announced November 2022.
-
On the exponent of cyclic codes
Authors:
Anuj Kumar Bhagat,
Ritumoni Sarma
Abstract:
We propose an algorithm to find a lower bound for the number of cyclic codes over any finite field with any given exponent. Besides, we give a formula to find the exponent of BCH codes.
We propose an algorithm to find a lower bound for the number of cyclic codes over any finite field with any given exponent. Besides, we give a formula to find the exponent of BCH codes.
△ Less
Submitted 31 August, 2022; v1 submitted 30 August, 2022;
originally announced August 2022.
-
Importance is in your attention: agent importance prediction for autonomous driving
Authors:
Christopher Hazard,
Akshay Bhagat,
Balarama Raju Buddharaju,
Zhongtao Liu,
Yunming Shao,
Lu Lu,
Sammy Omari,
Henggang Cui
Abstract:
Trajectory prediction is an important task in autonomous driving. State-of-the-art trajectory prediction models often use attention mechanisms to model the interaction between agents. In this paper, we show that the attention information from such models can also be used to measure the importance of each agent with respect to the ego vehicle's future planned trajectory. Our experiment results on t…
▽ More
Trajectory prediction is an important task in autonomous driving. State-of-the-art trajectory prediction models often use attention mechanisms to model the interaction between agents. In this paper, we show that the attention information from such models can also be used to measure the importance of each agent with respect to the ego vehicle's future planned trajectory. Our experiment results on the nuPlans dataset show that our method can effectively find and rank surrounding agents by their impact on the ego's plan.
△ Less
Submitted 19 April, 2022;
originally announced April 2022.
-
The ComMA Dataset V0.2: Annotating Aggression and Bias in Multilingual Social Media Discourse
Authors:
Ritesh Kumar,
Enakshi Nandi,
Laishram Niranjana Devi,
Shyam Ratan,
Siddharth Singh,
Akash Bhagat,
Yogesh Dawer
Abstract:
In this paper, we discuss the development of a multilingual dataset annotated with a hierarchical, fine-grained tagset marking different types of aggression and the "context" in which they occur. The context, here, is defined by the conversational thread in which a specific comment occurs and also the "type" of discursive role that the comment is performing with respect to the previous comment. Th…
▽ More
In this paper, we discuss the development of a multilingual dataset annotated with a hierarchical, fine-grained tagset marking different types of aggression and the "context" in which they occur. The context, here, is defined by the conversational thread in which a specific comment occurs and also the "type" of discursive role that the comment is performing with respect to the previous comment. The initial dataset, being discussed here (and made available as part of the ComMA@ICON shared task), consists of a total 15,000 annotated comments in four languages - Meitei, Bangla, Hindi, and Indian English - collected from various social media platforms such as YouTube, Facebook, Twitter and Telegram. As is usual on social media websites, a large number of these comments are multilingual, mostly code-mixed with English. The paper gives a detailed description of the tagset being used for annotation and also the process of developing a multi-label, fine-grained tagset that can be used for marking comments with aggression and bias of various kinds including gender bias, religious intolerance (called communal bias in the tagset), class/caste bias and ethnic/racial bias. We also define and discuss the tags that have been used for marking different the discursive role being performed through the comments, such as attack, defend, etc. We also present a statistical analysis of the dataset as well as results of our baseline experiments with developing an automatic aggression identification system using the dataset developed.
△ Less
Submitted 19 November, 2021;
originally announced November 2021.
-
Bayesian graph convolutional neural networks via tempered MCMC
Authors:
Rohitash Chandra,
Ayush Bhagat,
Manavendra Maharana,
Pavel N. Krivitsky
Abstract:
Deep learning models, such as convolutional neural networks, have long been applied to image and multi-media tasks, particularly those with structured data. More recently, there has been more attention to unstructured data that can be represented via graphs. These types of data are often found in health and medicine, social networks, and research data repositories. Graph convolutional neural netwo…
▽ More
Deep learning models, such as convolutional neural networks, have long been applied to image and multi-media tasks, particularly those with structured data. More recently, there has been more attention to unstructured data that can be represented via graphs. These types of data are often found in health and medicine, social networks, and research data repositories. Graph convolutional neural networks have recently gained attention in the field of deep learning that takes advantage of graph-based data representation with automatic feature extraction via convolutions. Given the popularity of these methods in a wide range of applications, robust uncertainty quantification is vital. This remains a challenge for large models and unstructured datasets. Bayesian inference provides a principled approach to uncertainty quantification of model parameters for deep learning models. Although Bayesian inference has been used extensively elsewhere, its application to deep learning remains limited due to the computational requirements of the Markov Chain Monte Carlo (MCMC) methods. Recent advances in parallel computing and advanced proposal schemes in MCMC sampling methods has opened the path for Bayesian deep learning. In this paper, we present Bayesian graph convolutional neural networks that employ tempered MCMC sampling with Langevin-gradient proposal distribution implemented via parallel computing. Our results show that the proposed method can provide accuracy similar to advanced optimisers while providing uncertainty quantification for key benchmark problems.
△ Less
Submitted 16 September, 2021; v1 submitted 17 April, 2021;
originally announced April 2021.
-
Developing a Multilingual Annotated Corpus of Misogyny and Aggression
Authors:
Shiladitya Bhattacharya,
Siddharth Singh,
Ritesh Kumar,
Akanksha Bansal,
Akash Bhagat,
Yogesh Dawer,
Bornini Lahiri,
Atul Kr. Ojha
Abstract:
In this paper, we discuss the development of a multilingual annotated corpus of misogyny and aggression in Indian English, Hindi, and Indian Bangla as part of a project on studying and automatically identifying misogyny and communalism on social media (the ComMA Project). The dataset is collected from comments on YouTube videos and currently contains a total of over 20,000 comments. The comments a…
▽ More
In this paper, we discuss the development of a multilingual annotated corpus of misogyny and aggression in Indian English, Hindi, and Indian Bangla as part of a project on studying and automatically identifying misogyny and communalism on social media (the ComMA Project). The dataset is collected from comments on YouTube videos and currently contains a total of over 20,000 comments. The comments are annotated at two levels - aggression (overtly aggressive, covertly aggressive, and non-aggressive) and misogyny (gendered and non-gendered). We describe the process of data collection, the tagset used for annotation, and issues and challenges faced during the process of annotation. Finally, we discuss the results of the baseline experiments conducted to develop a classifier for misogyny in the three languages.
△ Less
Submitted 16 March, 2020;
originally announced March 2020.