Showing 1–14 of 14 results for author: Paul, I

BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions

Authors: Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen Gong, Thong Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Yadav, Naman Jain, Alex Gu, Zhoujun Cheng, Jiawei Liu, Qian Liu , et al. (8 additional authors not shown)

Abstract: Automated software engineering has been greatly empowered by the recent advances in Large Language Models (LLMs) for programming. While current benchmarks have shown that LLMs can perform various software engineering tasks like human developers, the majority of their evaluations are limited to short and self-contained algorithmic tasks. Solving challenging and practical programming tasks requires… ▽ More Automated software engineering has been greatly empowered by the recent advances in Large Language Models (LLMs) for programming. While current benchmarks have shown that LLMs can perform various software engineering tasks like human developers, the majority of their evaluations are limited to short and self-contained algorithmic tasks. Solving challenging and practical programming tasks requires the capability of utilizing diverse function calls as tools to efficiently implement functionalities like data analysis and web development. In addition, using multiple tools to solve a task needs compositional reasoning by accurately understanding complex instructions. Fulfilling both of these characteristics can pose a great challenge for LLMs. To assess how well LLMs can solve challenging and practical programming tasks, we introduce Bench, a benchmark that challenges LLMs to invoke multiple function calls as tools from 139 libraries and 7 domains for 1,140 fine-grained programming tasks. To evaluate LLMs rigorously, each programming task encompasses 5.6 test cases with an average branch coverage of 99%. In addition, we propose a natural-language-oriented variant of Bench, Benchi, that automatically transforms the original docstrings into short instructions only with essential information. Our extensive evaluation of 60 LLMs shows that LLMs are not yet capable of following complex instructions to use function calls precisely, with scores up to 60%, significantly lower than the human performance of 97%. The results underscore the need for further advancements in this area. △ Less

Submitted 26 June, 2024; v1 submitted 22 June, 2024; originally announced June 2024.

Comments: 44 pages, 14 figures, 7 tables, built with love by the BigCode community :)

Bipartite powers of some classes of bipartite graphs

Authors: Indrajit Paul, Ashok Kumar Das

Abstract: Graph powers are a well-studied concept in graph theory. Analogous to graph powers, Chandran et al.[3] introduced the concept of bipartite powers for bipartite graphs. In this paper, we will demonstrate that some well-known classes of bipartite graphs, namely the interval bigraphs, proper interval bigraphs, and bigraphs of Ferrers dimension 2, are closed under the operation of taking bipartite pow… ▽ More Graph powers are a well-studied concept in graph theory. Analogous to graph powers, Chandran et al.[3] introduced the concept of bipartite powers for bipartite graphs. In this paper, we will demonstrate that some well-known classes of bipartite graphs, namely the interval bigraphs, proper interval bigraphs, and bigraphs of Ferrers dimension 2, are closed under the operation of taking bipartite powers. Finally, we define strongly closed property for bipartite graphs under powers and have shown that the class of chordal bipartite graphs is strongly closed under powers. △ Less

Submitted 26 April, 2024; originally announced April 2024.

IRCoder: Intermediate Representations Make Language Models Robust Multilingual Code Generators

Authors: Indraneil Paul, Goran Glavaš, Iryna Gurevych

Abstract: Code understanding and generation have fast become some of the most popular applications of language models (LMs). Nonetheless, research on multilingual aspects of Code-LMs (i.e., LMs for code generation) such as cross-lingual transfer between different programming languages, language-specific data augmentation, and post-hoc LM adaptation, alongside exploitation of data sources other than the orig… ▽ More Code understanding and generation have fast become some of the most popular applications of language models (LMs). Nonetheless, research on multilingual aspects of Code-LMs (i.e., LMs for code generation) such as cross-lingual transfer between different programming languages, language-specific data augmentation, and post-hoc LM adaptation, alongside exploitation of data sources other than the original textual content, has been much sparser than for their natural language counterparts. In particular, most mainstream Code-LMs have been pre-trained on source code files alone. In this work, we investigate the prospect of leveraging readily available compiler intermediate representations (IR) - shared across programming languages - to improve the multilingual capabilities of Code-LMs and facilitate cross-lingual transfer. To this end, we first compile SLTrans, a parallel dataset consisting of nearly 4M self-contained source code files coupled with respective intermediate representations. Next, starting from various base Code-LMs (ranging in size from 1.1B to 7.3B parameters), we carry out continued causal language modelling training on SLTrans, forcing the Code-LMs to (1) learn the IR language and (2) align the IR constructs with respective constructs of various programming languages. Our resulting models, dubbed IRCoder, display sizeable and consistent gains across a wide variety of code generation tasks and metrics, including prompt robustness, multilingual code completion, code understanding, and instruction following. △ Less

Submitted 15 April, 2024; v1 submitted 6 March, 2024; originally announced March 2024.

StarCoder 2 and The Stack v2: The Next Generation

Authors: Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, Zhuang Li, Wen-Ding Li, Megan Risdal, Jia Li, Jian Zhu, Terry Yue Zhuo , et al. (41 additional authors not shown)

Abstract: The BigCode project, an open-scientific collaboration focused on the responsible development of Large Language Models for Code (Code LLMs), introduces StarCoder2. In partnership with Software Heritage (SWH), we build The Stack v2 on top of the digital commons of their source code archive. Alongside the SWH repositories spanning 619 programming languages, we carefully select other high-quality data… ▽ More The BigCode project, an open-scientific collaboration focused on the responsible development of Large Language Models for Code (Code LLMs), introduces StarCoder2. In partnership with Software Heritage (SWH), we build The Stack v2 on top of the digital commons of their source code archive. Alongside the SWH repositories spanning 619 programming languages, we carefully select other high-quality data sources, such as GitHub pull requests, Kaggle notebooks, and code documentation. This results in a training set that is 4x larger than the first StarCoder dataset. We train StarCoder2 models with 3B, 7B, and 15B parameters on 3.3 to 4.3 trillion tokens and thoroughly evaluate them on a comprehensive set of Code LLM benchmarks. We find that our small model, StarCoder2-3B, outperforms other Code LLMs of similar size on most benchmarks, and also outperforms StarCoderBase-15B. Our large model, StarCoder2- 15B, significantly outperforms other models of comparable size. In addition, it matches or outperforms CodeLlama-34B, a model more than twice its size. Although DeepSeekCoder- 33B is the best-performing model at code completion for high-resource languages, we find that StarCoder2-15B outperforms it on math and code reasoning benchmarks, as well as several low-resource languages. We make the model weights available under an OpenRAIL license and ensure full transparency regarding the training data by releasing the SoftWare Heritage persistent IDentifiers (SWHIDs) of the source code data. △ Less

Submitted 29 February, 2024; originally announced February 2024.

SRAM Alpha-SER Estimation From Word-Line Voltage Margin Measurements: Design Architecture and Experimental Results

Authors: Gabriel Torrens, Ivan de Paul, Bartomeu Alorda, Sebastia Bota, Jaume Segura

Abstract: Experimental results from a 65 nm CMOS commercial technology SRAM test chip reveal a linear correlation between a new electrical parameter -- the word-line voltage margin (VWLVM) -- and the measured circuit alpha-SER. Additional experiments show that no other memory cell electrical robustness-related parameters exhibit such correlation. The technique proposed is based on correlating the VWLVM to t… ▽ More Experimental results from a 65 nm CMOS commercial technology SRAM test chip reveal a linear correlation between a new electrical parameter -- the word-line voltage margin (VWLVM) -- and the measured circuit alpha-SER. Additional experiments show that no other memory cell electrical robustness-related parameters exhibit such correlation. The technique proposed is based on correlating the VWLVM to the SER measured on a small number of circuit samples to determine the correlation parameters. Then, the remaining non-irradiated circuits SER is determined from electrical measurements (VWLVM) without the need of additional radiation experiments. This method represents a significant improvement in time and cost, while simplifying the SER-determination methods since most of the circuits do not require irradiation. The technique involves a minor memory design modification that does not degrade circuit performance, while circuit area increase is negligible. △ Less

Submitted 20 January, 2024; originally announced February 2024.

Comments: 6 pages, 10 figures

Journal ref: IEEE Transactions on Nuclear Science, vol. 61, no. 4, pp. 1849-1855, Aug. 2014

Adapters: A Unified Library for Parameter-Efficient and Modular Transfer Learning

Authors: Clifton Poth, Hannah Sterz, Indraneil Paul, Sukannya Purkayastha, Leon Engländer, Timo Imhof, Ivan Vulić, Sebastian Ruder, Iryna Gurevych, Jonas Pfeiffer

Abstract: We introduce Adapters, an open-source library that unifies parameter-efficient and modular transfer learning in large language models. By integrating 10 diverse adapter methods into a unified interface, Adapters offers ease of use and flexible configuration. Our library allows researchers and practitioners to leverage adapter modularity through composition blocks, enabling the design of complex ad… ▽ More We introduce Adapters, an open-source library that unifies parameter-efficient and modular transfer learning in large language models. By integrating 10 diverse adapter methods into a unified interface, Adapters offers ease of use and flexible configuration. Our library allows researchers and practitioners to leverage adapter modularity through composition blocks, enabling the design of complex adapter setups. We demonstrate the library's efficacy by evaluating its performance against full fine-tuning on various NLP tasks. Adapters provides a powerful tool for addressing the challenges of conventional fine-tuning paradigms and promoting more efficient and modular transfer learning. The library is available via https://adapterhub.ml/adapters. △ Less

Submitted 18 November, 2023; originally announced November 2023.

Comments: EMNLP 2023: Systems Demonstrations

Obstruction characterization of co-TT graphs

Authors: Ashok Kumar Das, Indrajit Paul

Abstract: Threshold tolerance graphs and their complement graphs ( known as co-TT graphs) were introduced by Monma, Reed and Trotter[24]. Introducing the concept of negative interval Hell et al.[19] defined signed-interval bigraphs/digraphs and have shown that they are equivalent to several seemingly different classes of bigraphs/digraphs. They have also shown that co-TT graphs are equivalent to symmetric s… ▽ More Threshold tolerance graphs and their complement graphs ( known as co-TT graphs) were introduced by Monma, Reed and Trotter[24]. Introducing the concept of negative interval Hell et al.[19] defined signed-interval bigraphs/digraphs and have shown that they are equivalent to several seemingly different classes of bigraphs/digraphs. They have also shown that co-TT graphs are equivalent to symmetric signed-interval digraphs. In this paper we characterize signed-interval bigraphs and signed-interval graphs respectively in terms of their biadjacency matrices and adjacency matrices. Finally, based on the geometric representation of signed-interval graphs we have setteled the open problem of forbidden induced subgraph characterization of co-TT graphs posed by Monma, Reed and Trotter in the same paper. △ Less

Submitted 24 August, 2023; originally announced August 2023.

Comments: arXiv admin note: substantial text overlap with arXiv:2206.05917

FACTIFY3M: A Benchmark for Multimodal Fact Verification with Explainability through 5W Question-Answering

Authors: Megha Chakraborty, Khushbu Pahwa, Anku Rani, Shreyas Chatterjee, Dwip Dalal, Harshit Dave, Ritvik G, Preethi Gurumurthy, Adarsh Mahor, Samahriti Mukherjee, Aditya Pakala, Ishan Paul, Janvita Reddy, Arghya Sarkar, Kinjal Sensharma, Aman Chadha, Amit P. Sheth, Amitava Das

Abstract: Combating disinformation is one of the burning societal crises -- about 67% of the American population believes that disinformation produces a lot of uncertainty, and 10% of them knowingly propagate disinformation. Evidence shows that disinformation can manipulate democratic processes and public opinion, causing disruption in the share market, panic and anxiety in society, and even death during cr… ▽ More Combating disinformation is one of the burning societal crises -- about 67% of the American population believes that disinformation produces a lot of uncertainty, and 10% of them knowingly propagate disinformation. Evidence shows that disinformation can manipulate democratic processes and public opinion, causing disruption in the share market, panic and anxiety in society, and even death during crises. Therefore, disinformation should be identified promptly and, if possible, mitigated. With approximately 3.2 billion images and 720,000 hours of video shared online daily on social media platforms, scalable detection of multimodal disinformation requires efficient fact verification. Despite progress in automatic text-based fact verification (e.g., FEVER, LIAR), the research community lacks substantial effort in multimodal fact verification. To address this gap, we introduce FACTIFY 3M, a dataset of 3 million samples that pushes the boundaries of the domain of fact verification via a multimodal fake news dataset, in addition to offering explainability through the concept of 5W question-answering. Salient features of the dataset include: (i) textual claims, (ii) ChatGPT-generated paraphrased claims, (iii) associated images, (iv) stable diffusion-generated additional images (i.e., visual paraphrases), (v) pixel-level image heatmap to foster image-text explainability of the claim, (vi) 5W QA pairs, and (vii) adversarial fake news stories. △ Less

Submitted 30 October, 2023; v1 submitted 22 May, 2023; originally announced June 2023.

Comments: arXiv admin note: text overlap with arXiv:2305.04329

On powers of circular arc graphs

Authors: Ashok Kumar Das, Indrajit Paul

Abstract: A class of graphs $\mathcal{C}$ is closed under powers if for every graph $G\in\mathcal{C}$ and every $k\in\mathbb{N}$, $G^k\in\mathcal{C}$. Also $\mathcal{C}$ is strongly closed under powers if for every $k\in\mathbb{N}$, if $G^k\in\mathcal{C}$, then $G^{k+1}\in\mathcal{C}$. It is known that circular arc graphs and proper circular arc graphs are closed under powers. But it is open whether these c… ▽ More A class of graphs $\mathcal{C}$ is closed under powers if for every graph $G\in\mathcal{C}$ and every $k\in\mathbb{N}$, $G^k\in\mathcal{C}$. Also $\mathcal{C}$ is strongly closed under powers if for every $k\in\mathbb{N}$, if $G^k\in\mathcal{C}$, then $G^{k+1}\in\mathcal{C}$. It is known that circular arc graphs and proper circular arc graphs are closed under powers. But it is open whether these classes of graphs are also strongly closed under powers. In this paper we have settled these problems. △ Less

Submitted 8 June, 2023; v1 submitted 17 October, 2022; originally announced October 2022.

Signed interval graphs and bigraphs: A generalization of interval graphs and bigraphs

Authors: Ashok Kumar Das, Indrajit Paul

Abstract: In this paper, we define and characterize signed interval graphs and bigraphs introducing the concept of negative interval. Also we have shown that these classes of graphs are respectively a generalization of well known classes of interval graphs and interval bigraphs. In this context we have observed that signed interval graphs coincide with the complement of Threshold tolerance graphs(co-TT grap… ▽ More In this paper, we define and characterize signed interval graphs and bigraphs introducing the concept of negative interval. Also we have shown that these classes of graphs are respectively a generalization of well known classes of interval graphs and interval bigraphs. In this context we have observed that signed interval graphs coincide with the complement of Threshold tolerance graphs(co-TT graphs) introduced by Monma, Reed and Trotter \cite{22}. Finally, we have solved the open problem of forbidden induced subgraph characterization of co-TT graphs posed by them in the same paper. △ Less

Submitted 13 June, 2022; originally announced June 2022.

On Hamiltonian-Connected and Mycielski graphs

Authors: Ashok Kumar Das, Indrajit Paul

Abstract: A graph $G$ is Hamiltonian-connected if there exists a Hamiltonian path between any two vertices of $G$. It is known that if $G$ is 2-connected then the graph $G^2$ is Hamiltonian-connected. In this paper we prove that the square of every self-complementary graph of order grater than 4 is Hamiltonian-connected. If $G$ is a $k$-critical graph, then we prove that the Mycielski graph $μ(G)$ is… ▽ More A graph $G$ is Hamiltonian-connected if there exists a Hamiltonian path between any two vertices of $G$. It is known that if $G$ is 2-connected then the graph $G^2$ is Hamiltonian-connected. In this paper we prove that the square of every self-complementary graph of order grater than 4 is Hamiltonian-connected. If $G$ is a $k$-critical graph, then we prove that the Mycielski graph $μ(G)$ is $(k+1)$-critical graph. Jarnicki et al.[7] proved that for every Hamiltonian graph of odd order, the Mycielski graph $μ(G)$ of $G$ is Hamiltonian-connected. They also pose a conjecture that if $G$ is Hamiltonian-connected and not $K_2$ then $μ(G)$ is Hamiltonian-connected. In this paper we also prove this conjecture. △ Less

Submitted 6 February, 2023; v1 submitted 6 June, 2022; originally announced June 2022.

What sets Verified Users apart? Insights, Analysis and Prediction of Verified Users on Twitter

Authors: Indraneil Paul, Abhinav Khattar, Shaan Chopra, Ponnurangam Kumaraguru, Manish Gupta

Abstract: Social network and publishing platforms, such as Twitter, support the concept of a secret proprietary verification process, for handles they deem worthy of platform-wide public interest. In line with significant prior work which suggests that possessing such a status symbolizes enhanced credibility in the eyes of the platform audience, a verified badge is clearly coveted among public figures and b… ▽ More Social network and publishing platforms, such as Twitter, support the concept of a secret proprietary verification process, for handles they deem worthy of platform-wide public interest. In line with significant prior work which suggests that possessing such a status symbolizes enhanced credibility in the eyes of the platform audience, a verified badge is clearly coveted among public figures and brands. What are less obvious are the inner workings of the verification process and what being verified represents. This lack of clarity, coupled with the flak that Twitter received by extending aforementioned status to political extremists in 2017, backed Twitter into publicly admitting that the process and what the status represented needed to be rethought. With this in mind, we seek to unravel the aspects of a user's profile which likely engender or preclude verification. The aim of the paper is two-fold: First, we test if discerning the verification status of a handle from profile metadata and content features is feasible. Second, we unravel the features which have the greatest bearing on a handle's verification status. We collected a dataset consisting of profile metadata of all 231,235 verified English-speaking users (as of July 2018), a control sample of 175,930 non-verified English-speaking users and all their 494 million tweets over a one year collection period. Our proposed models are able to reliably identify verification status (Area under curve AUC > 99%). We show that number of public list memberships, presence of neutral sentiment in tweets and an authoritative language style are the most pertinent predictors of verification status. To the best of our knowledge, this work represents the first attempt at discerning and classifying verification worthy users on Twitter. △ Less

Submitted 12 March, 2019; originally announced March 2019.

Elites Tweet? Characterizing the Twitter Verified User Network

Authors: Indraneil Paul, Abhinav Khattar, Ponnurangam Kumaraguru, Manish Gupta, Shaan Chopra

Abstract: Social network and publishing platforms, such as Twitter, support the concept of verification. Verified accounts are deemed worthy of platform-wide public interest and are separately authenticated by the platform itself. There have been repeated assertions by these platforms about verification not being tantamount to endorsement. However, a significant body of prior work suggests that possessing a… ▽ More Social network and publishing platforms, such as Twitter, support the concept of verification. Verified accounts are deemed worthy of platform-wide public interest and are separately authenticated by the platform itself. There have been repeated assertions by these platforms about verification not being tantamount to endorsement. However, a significant body of prior work suggests that possessing a verified status symbolizes enhanced credibility in the eyes of the platform audience. As a result, such a status is highly coveted among public figures and influencers. Hence, we attempt to characterize the network of verified users on Twitter and compare the results to similar analysis performed for the entire Twitter network. We extracted the entire network of verified users on Twitter (as of July 2018) and obtained 231,246 user profiles and 79,213,811 connections. Subsequently in the network analysis, we found that the sub-graph of verified users mirrors the full Twitter users graph in some aspects such as possessing a short diameter. However, our findings contrast with earlier findings on multiple aspects, such as the possession of a power law out-degree distribution, slight dissortativity and a significantly higher reciprocity rate, as elucidated in the paper. Moreover, we attempt to gauge the presence of salient components within this sub-graph and detect the absence of homophily with respect to popularity, which again is in stark contrast to the full Twitter graph. Finally, we demonstrate stationarity in the time series of verified user activity levels. To the best of our knowledge, this work represents the first quantitative attempt at characterizing verified users on Twitter. △ Less

Submitted 12 March, 2019; v1 submitted 23 December, 2018; originally announced December 2018.

Implications of Integrated CPU-GPU Processors on Thermal and Power Management Techniques

Authors: Kapil Dev, Indrani Paul, Wei Huang, Yasuko Eckert, Wayne Burleson, Sherief Reda

Abstract: Heterogeneous processors with architecturally different cores (CPU and GPU) integrated on the same die lead to new challenges and opportunities for thermal and power management techniques because of shared thermal/power budgets between these cores. In this paper, we show that new parallel programming paradigms (e.g., OpenCL) for CPU-GPU processors create a tighter coupling between the workload, th… ▽ More Heterogeneous processors with architecturally different cores (CPU and GPU) integrated on the same die lead to new challenges and opportunities for thermal and power management techniques because of shared thermal/power budgets between these cores. In this paper, we show that new parallel programming paradigms (e.g., OpenCL) for CPU-GPU processors create a tighter coupling between the workload, the thermal/power management unit and the operating system. Using detailed thermal and power maps of the die from infrared imaging, we demonstrate that in contrast to traditional multi-core CPUs, heterogeneous processors exhibit higher coupled behavior for dynamic voltage and frequency scaling and workload scheduling, in terms of their effect on performance, power, and temperature. Further, we show that by taking the differences in core architectures and relative proximity of different computing cores on the die into consideration, better scheduling schemes could be implemented to reduce both the power density and peak temperature of the die. The findings presented in the paper can be used to improve thermal and power efficiency of heterogeneous CPU-GPU processors. △ Less

Submitted 29 August, 2018; originally announced August 2018.

Comments: 9 pages, 8 figures, 2 tables