-
Scaling Granite Code Models to 128K Context
Authors:
Matt Stallone,
Vaibhav Saxena,
Leonid Karlinsky,
Bridget McGinn,
Tim Bula,
Mayank Mishra,
Adriana Meza Soria,
Gaoyuan Zhang,
Aditya Prasad,
Yikang Shen,
Saptha Surendran,
Shanmukha Guttula,
Hima Patel,
Parameswaran Selvam,
Xuan-Hong Dang,
Yan Koyfman,
Atin Sood,
Rogerio Feris,
Nirmit Desai,
David D. Cox,
Ruchir Puri,
Rameswar Panda
Abstract:
This paper introduces long-context Granite code models that support effective context windows of up to 128K tokens. Our solution for scaling context length of Granite 3B/8B code models from 2K/4K to 128K consists of a light-weight continual pretraining by gradually increasing its RoPE base frequency with repository-level file packing and length-upsampled long-context data. Additionally, we also re…
▽ More
This paper introduces long-context Granite code models that support effective context windows of up to 128K tokens. Our solution for scaling context length of Granite 3B/8B code models from 2K/4K to 128K consists of a light-weight continual pretraining by gradually increasing its RoPE base frequency with repository-level file packing and length-upsampled long-context data. Additionally, we also release instruction-tuned models with long-context support which are derived by further finetuning the long context base models on a mix of permissively licensed short and long-context instruction-response pairs. While comparing to the original short-context Granite code models, our long-context models achieve significant improvements on long-context tasks without any noticeable performance degradation on regular code completion benchmarks (e.g., HumanEval). We release all our long-context Granite code models under an Apache 2.0 license for both research and commercial use.
△ Less
Submitted 18 July, 2024;
originally announced July 2024.
-
Granite Code Models: A Family of Open Foundation Models for Code Intelligence
Authors:
Mayank Mishra,
Matt Stallone,
Gaoyuan Zhang,
Yikang Shen,
Aditya Prasad,
Adriana Meza Soria,
Michele Merler,
Parameswaran Selvam,
Saptha Surendran,
Shivdeep Singh,
Manish Sethi,
Xuan-Hong Dang,
Pengyuan Li,
Kun-Lung Wu,
Syed Zawad,
Andrew Coleman,
Matthew White,
Mark Lewis,
Raju Pavuluri,
Yan Koyfman,
Boris Lublinsky,
Maximilien de Bayser,
Ibrahim Abdelaziz,
Kinjal Basu,
Mayank Agarwal
, et al. (21 additional authors not shown)
Abstract:
Large Language Models (LLMs) trained on code are revolutionizing the software development process. Increasingly, code LLMs are being integrated into software development environments to improve the productivity of human programmers, and LLM-based agents are beginning to show promise for handling complex tasks autonomously. Realizing the full potential of code LLMs requires a wide range of capabili…
▽ More
Large Language Models (LLMs) trained on code are revolutionizing the software development process. Increasingly, code LLMs are being integrated into software development environments to improve the productivity of human programmers, and LLM-based agents are beginning to show promise for handling complex tasks autonomously. Realizing the full potential of code LLMs requires a wide range of capabilities, including code generation, fixing bugs, explaining and documenting code, maintaining repositories, and more. In this work, we introduce the Granite series of decoder-only code models for code generative tasks, trained with code written in 116 programming languages. The Granite Code models family consists of models ranging in size from 3 to 34 billion parameters, suitable for applications ranging from complex application modernization tasks to on-device memory-constrained use cases. Evaluation on a comprehensive set of tasks demonstrates that Granite Code models consistently reaches state-of-the-art performance among available open-source code LLMs. The Granite Code model family was optimized for enterprise software development workflows and performs well across a range of coding tasks (e.g. code generation, fixing and explanation), making it a versatile all around code model. We release all our Granite Code models under an Apache 2.0 license for both research and commercial use.
△ Less
Submitted 7 May, 2024;
originally announced May 2024.
-
A Survey on Solving and Discovering Differential Equations Using Deep Neural Networks
Authors:
Hyeonjung,
Jung,
Jayant Gupta,
Bharat Jayaprakash,
Matthew Eagon,
Harish Panneer Selvam,
Carl Molnar,
William Northrop,
Shashi Shekhar
Abstract:
Ordinary and partial differential equations (DE) are used extensively in scientific and mathematical domains to model physical systems. Current literature has focused primarily on deep neural network (DNN) based methods for solving a specific DE or a family of DEs. Research communities with a history of using DE models may view DNN-based differential equation solvers (DNN-DEs) as a faster and tran…
▽ More
Ordinary and partial differential equations (DE) are used extensively in scientific and mathematical domains to model physical systems. Current literature has focused primarily on deep neural network (DNN) based methods for solving a specific DE or a family of DEs. Research communities with a history of using DE models may view DNN-based differential equation solvers (DNN-DEs) as a faster and transferable alternative to current numerical methods. However, there is a lack of systematic surveys detailing the use of DNN-DE methods across physical application domains and a generalized taxonomy to guide future research. This paper surveys and classifies previous works and provides an educational tutorial for senior practitioners, professionals, and graduate students in engineering and computer science. First, we propose a taxonomy to navigate domains of DE systems studied under the umbrella of DNN-DE. Second, we examine the theory and performance of the Physics Informed Neural Network (PINN) to demonstrate how the influential DNN-DE architecture mathematically solves a system of equations. Third, to reinforce the key ideas of solving and discovery of DEs using DNN, we provide a tutorial using DeepXDE, a Python package for developing PINNs, to develop DNN-DEs for solving and discovering a classic DE, the linear transport equation.
△ Less
Submitted 19 June, 2023; v1 submitted 26 April, 2023;
originally announced April 2023.
-
DIPPM: a Deep Learning Inference Performance Predictive Model using Graph Neural Networks
Authors:
Karthick Panner Selvam,
Mats Brorsson
Abstract:
Deep Learning (DL) has developed to become a corner-stone in many everyday applications that we are now relying on. However, making sure that the DL model uses the underlying hardware efficiently takes a lot of effort. Knowledge about inference characteristics can help to find the right match so that enough resources are given to the model, but not too much. We have developed a DL Inference Perfor…
▽ More
Deep Learning (DL) has developed to become a corner-stone in many everyday applications that we are now relying on. However, making sure that the DL model uses the underlying hardware efficiently takes a lot of effort. Knowledge about inference characteristics can help to find the right match so that enough resources are given to the model, but not too much. We have developed a DL Inference Performance Predictive Model (DIPPM) that predicts the inference latency, energy, and memory usage of a given input DL model on the NVIDIA A100 GPU. We also devised an algorithm to suggest the appropriate A100 Multi-Instance GPU profile from the output of DIPPM. We developed a methodology to convert DL models expressed in multiple frameworks to a generalized graph structure that is used in DIPPM. It means DIPPM can parse input DL models from various frameworks. Our DIPPM can be used not only helps to find suitable hardware configurations but also helps to perform rapid design-space exploration for the inference performance of a model. We constructed a graph multi-regression dataset consisting of 10,508 different DL models to train and evaluate the performance of DIPPM, and reached a resulting Mean Absolute Percentage Error (MAPE) as low as 1.9%.
△ Less
Submitted 21 March, 2023;
originally announced March 2023.
-
PMNet: Robust Pathloss Map Prediction via Supervised Learning
Authors:
Ju-Hyung Lee,
Omer Gokalp Serbetci,
Dheeraj Panneer Selvam,
Andreas F. Molisch
Abstract:
Pathloss prediction is an essential component of wireless network planning. While ray tracing based methods have been successfully used for many years, they require significant computational effort that may become prohibitive with the increased network densification and/or use of higher frequencies in 5G/B5G (beyond 5G) systems. In this paper, we propose and evaluate a data-driven and model-free p…
▽ More
Pathloss prediction is an essential component of wireless network planning. While ray tracing based methods have been successfully used for many years, they require significant computational effort that may become prohibitive with the increased network densification and/or use of higher frequencies in 5G/B5G (beyond 5G) systems. In this paper, we propose and evaluate a data-driven and model-free pathloss prediction method, dubbed PMNet. This method uses a supervised learning approach: training a neural network (NN) with a limited amount of ray tracing (or channel measurement) data and map data and then predicting the pathloss over location with no ray tracing data with a high level of accuracy. Our proposed pathloss map prediction-oriented NN architecture, which is empowered by state-of-the-art computer vision techniques, outperforms other architectures that have been previously proposed (e.g., UNet, RadioUNet) in terms of accuracy while showing generalization capability. Moreover, PMNet trained on a 4-fold smaller dataset surpasses the other baselines (trained on a 4-fold larger dataset), corroborating the potential of PMNet.
△ Less
Submitted 16 May, 2023; v1 submitted 18 November, 2022;
originally announced November 2022.
-
Vehicle Emissions Prediction with Physics-Aware AI Models: Preliminary Results
Authors:
Harish Panneer Selvam,
Yan Li,
Pengyue Wang,
William F. Northrop,
Shashi Shekhar
Abstract:
Given an on-board diagnostics (OBD) dataset and a physics-based emissions prediction model, this paper aims to develop an accurate and computational-efficient AI (Artificial Intelligence) method that predicts vehicle emissions. The problem is of societal importance because vehicular emissions lead to climate change and impact human health. This problem is challenging because the OBD data does not…
▽ More
Given an on-board diagnostics (OBD) dataset and a physics-based emissions prediction model, this paper aims to develop an accurate and computational-efficient AI (Artificial Intelligence) method that predicts vehicle emissions. The problem is of societal importance because vehicular emissions lead to climate change and impact human health. This problem is challenging because the OBD data does not contain enough parameters needed by high-order physics models. Conversely, related work has shown that low-order physics models have poor predictive accuracy when using available OBD data. This paper uses a divergent window co-occurrence pattern detection method to develop a spatiotemporal variability-aware AI model for predicting emission values from the OBD datasets. We conducted a case study using real-world OBD data from a local public transportation agency. Results show that the proposed AI method has approximately 65% improved predictive accuracy than a non-AI low-order physics model and is approximately 35% more accurate than a baseline model.
△ Less
Submitted 1 May, 2021;
originally announced May 2021.
-
Blockchain Based Accounts Payable Platform for Goods Trade
Authors:
Krishnasuri Narayanam,
Seep Goel,
Abhishek Singh,
Yedendra Shrinivasan,
Parameswaram Selvam
Abstract:
Goods trade is a supply chain transaction that involves shippers buying goods from suppliers and carriers providing goods transportation. Shippers are issued invoices from suppliers and carriers. Shippers carry out goods receiving and invoice processing before payment processing of bills for suppliers and carriers, where invoice processing includes tasks like processing claims and adjusting the bi…
▽ More
Goods trade is a supply chain transaction that involves shippers buying goods from suppliers and carriers providing goods transportation. Shippers are issued invoices from suppliers and carriers. Shippers carry out goods receiving and invoice processing before payment processing of bills for suppliers and carriers, where invoice processing includes tasks like processing claims and adjusting the bill payments. Goods receiving involves verification of received goods by the Shipper's receiving team. Invoice processing is carried out by the Shipper's accounts payable team, which in turn is verified by the accounts receivable teams of suppliers and carriers. This paper presents a blockchain-based accounts payable system that generates claims for the deficiency in the goods received and accordingly adjusts the payment in the bills for suppliers and carriers. Primary motivations for these supply chain organizations to adopt blockchain-based accounts payable systems are to eliminate the process redundancies (accounts payable vs. accounts receivable), to reduce the number of disputes among the transacting participants, and to accelerate the accounts payable processes via optimizations in the claims generation and blockchain-based dispute reconciliation.
△ Less
Submitted 4 March, 2021;
originally announced March 2021.
-
Circuit Complexity From Supersymmetric Quantum Field Theory With Morse Function
Authors:
Sayantan Choudhury,
Sachin Panneer Selvam,
K. Shirish
Abstract:
Computation of circuit complexity has gained much attention in the Theoretical Physics community in recent times to gain insights into the chaotic features and random fluctuations of fields in the quantum regime. Recent studies of circuit complexity take inspiration from Nielsen's geometric approach, which is based on the idea of optimal quantum control in which a cost function is introduced for t…
▽ More
Computation of circuit complexity has gained much attention in the Theoretical Physics community in recent times to gain insights into the chaotic features and random fluctuations of fields in the quantum regime. Recent studies of circuit complexity take inspiration from Nielsen's geometric approach, which is based on the idea of optimal quantum control in which a cost function is introduced for the various possible path to determine the optimum circuit. In this paper, we study the relationship between the circuit complexity and Morse theory within the framework of algebraic topology, which will then help us study circuit complexity in supersymmetric quantum field theory describing both simple and inverted harmonic oscillators up to higher orders of quantum corrections. We will restrict ourselves to $\mathcal{N} = 1$ supersymmetry with one fermionic generator $Q_α$. The expression of circuit complexity in quantum regime would then be given by the Hessian of the Morse function in supersymmetric quantum field theory. We also provide technical proof of the well known universal connecting relation between quantum chaos and circuit complexity of the supersymmetric quantum field theories, using the general description of Morse theory.
△ Less
Submitted 11 August, 2022; v1 submitted 25 January, 2021;
originally announced January 2021.
-
Circuit Complexity From Cosmological Islands
Authors:
Sayantan Choudhury,
Satyaki Chowdhury,
Nitin Gupta,
Anurag Mishara,
Sachin Panneer Selvam,
Sudhakar Panda,
Gabriel D. Pasquino,
Chiranjeeb Singha,
Abinash Swain
Abstract:
Recently in various theoretical works, path-breaking progress has been made in recovering the well-known Page Curve of an evaporating black hole with Quantum Extremal Islands, proposed to solve the long-standing black hole information loss problem related to the unitarity issue. Motivated by this concept, in this paper, we study cosmological circuit complexity in the presence (or absence) of Quant…
▽ More
Recently in various theoretical works, path-breaking progress has been made in recovering the well-known Page Curve of an evaporating black hole with Quantum Extremal Islands, proposed to solve the long-standing black hole information loss problem related to the unitarity issue. Motivated by this concept, in this paper, we study cosmological circuit complexity in the presence (or absence) of Quantum Extremal Islands in the negative (or positive) Cosmological Constant with radiation in the background of Friedmann-Lema$\hat{i}$tre-Robertson-Walker (FLRW) space-time i.e the presence and absence of islands in anti-de Sitter and the de Sitter spacetime having SO(2, 3) and SO(1, 4) isometries respectively. Without using any explicit details of any gravity model, we study the behaviour of the circuit complexity function with respect to the dynamical cosmological solution for the scale factors for the above-mentioned two situations in FLRW space-time using squeezed state formalism. By studying the cosmological circuit complexity, Out-of-Time Ordered Correlators, and entanglement entropy of the modes of the squeezed state, in different parameter spaces, we conclude the non-universality of these measures. Their remarkably different features in the different parameter spaces suggest their dependence on the parameters of the model under consideration.
△ Less
Submitted 3 July, 2021; v1 submitted 16 December, 2020;
originally announced December 2020.
-
Quantum aspects of chaos and complexity from bouncing cosmology: A study with two-mode single field squeezed state formalism
Authors:
Parth Bhargava,
Sayantan Choudhury,
Satyaki Chowdhury,
Anurag Mishara,
Sachin Panneer Selvam,
Sudhakar Panda,
Gabriel D. Pasquino
Abstract:
$Circuit~ Complexity$, a well known computational technique has recently become the backbone of the physics community to probe the chaotic behaviour and random quantum fluctuations of quantum fields. This paper is devoted to the study of out-of-equilibrium aspects and quantum chaos appearing in the universe from the paradigm of two well known bouncing cosmological solutions viz. $Cosine~ hyperboli…
▽ More
$Circuit~ Complexity$, a well known computational technique has recently become the backbone of the physics community to probe the chaotic behaviour and random quantum fluctuations of quantum fields. This paper is devoted to the study of out-of-equilibrium aspects and quantum chaos appearing in the universe from the paradigm of two well known bouncing cosmological solutions viz. $Cosine~ hyperbolic$ and $Exponential$ models of scale factors. Besides $circuit~ complexity$, we use the $Out-of-Time~ Ordered~ correlation~ (OTOC)$ functions for probing the random behaviour of the universe both at early and the late times. In particular, we use the techniques of well known two-mode squeezed state formalism in cosmological perturbation theory as a key ingredient for the purpose of our computation. To give an appropriate theoretical interpretation that is consistent with the observational perspective we use the scale factor and the number of e-foldings as a dynamical variable instead of conformal time for this computation. From this study, we found that the period of post bounce is the most interesting one. Though it may not be immediately visible, but an exponential rise can be seen in the $complexity$ once the post bounce feature is extrapolated to the present time scales. We also find within the very small acceptable error range a universal connecting relation between Complexity computed from two different kinds of cost functionals-$linearly~ weighted$ and $geodesic~ weighted$ with the OTOC. Furthermore, from the $complexity$ computation obtained from both the cosmological models and also using the well known MSS bound on quantum Lyapunov exponent, $λ\leq 2π/β$ for the saturation of chaos, we estimate the lower bound on the equilibrium temperature of our universe at late time scale. Finally, we provide a rough estimation of the scrambling time in terms of the conformal time.
△ Less
Submitted 15 September, 2021; v1 submitted 8 September, 2020;
originally announced September 2020.
