Showing 1–10 of 10 results for author: Mathews, N S

Whodunit: Classifying Code as Human Authored or GPT-4 Generated -- A case study on CodeChef problems

Authors: Oseremen Joy Idialu, Noble Saji Mathews, Rungroj Maipradit, Joanne M. Atlee, Mei Nagappan

Abstract: Artificial intelligence (AI) assistants such as GitHub Copilot and ChatGPT, built on large language models like GPT-4, are revolutionizing how programming tasks are performed, raising questions about whether code is authored by generative AI models. Such questions are of particular interest to educators, who worry that these tools enable a new form of academic dishonesty, in which students submit… ▽ More Artificial intelligence (AI) assistants such as GitHub Copilot and ChatGPT, built on large language models like GPT-4, are revolutionizing how programming tasks are performed, raising questions about whether code is authored by generative AI models. Such questions are of particular interest to educators, who worry that these tools enable a new form of academic dishonesty, in which students submit AI generated code as their own work. Our research explores the viability of using code stylometry and machine learning to distinguish between GPT-4 generated and human-authored code. Our dataset comprises human-authored solutions from CodeChef and AI-authored solutions generated by GPT-4. Our classifier outperforms baselines, with an F1-score and AUC-ROC score of 0.91. A variant of our classifier that excludes gameable features (e.g., empty lines, whitespace) still performs well with an F1-score and AUC-ROC score of 0.89. We also evaluated our classifier with respect to the difficulty of the programming problem and found that there was almost no difference between easier and intermediate problems, and the classifier performed only slightly worse on harder problems. Our study shows that code stylometry is a promising approach for distinguishing between GPT-4 generated code and human-authored code. △ Less

Submitted 6 March, 2024; originally announced March 2024.

Comments: 13 pages, 5 figures, MSR Conference

Test-Driven Development for Code Generation

Authors: Noble Saji Mathews, Meiyappan Nagappan

Abstract: Recent Large Language Models (LLMs) have demonstrated significant capabilities in generating code snippets directly from problem statements. This increasingly automated process mirrors traditional human-led software development, where code is often written in response to a requirement. Historically, Test-Driven Development (TDD) has proven its merit, requiring developers to write tests before the… ▽ More Recent Large Language Models (LLMs) have demonstrated significant capabilities in generating code snippets directly from problem statements. This increasingly automated process mirrors traditional human-led software development, where code is often written in response to a requirement. Historically, Test-Driven Development (TDD) has proven its merit, requiring developers to write tests before the functional code, ensuring alignment with the initial problem statements. Applying TDD principles to LLM-based code generation offers one distinct benefit: it enables developers to verify the correctness of generated code against predefined tests. This paper investigates if and how TDD can be incorporated into AI-assisted code-generation processes. We experimentally evaluate our hypothesis that providing LLMs like GPT-4 and Llama 3 with tests in addition to the problem statements enhances code generation outcomes. We experimented with established function-level code generation benchmarks such as MBPP and HumanEval. Our results consistently demonstrate that including test cases leads to higher success in solving programming challenges. We assert that TDD is a promising paradigm for helping ensure that the code generated by LLMs effectively captures the requirements. △ Less

Submitted 11 June, 2024; v1 submitted 20 February, 2024; originally announced February 2024.

FuzzSlice: Pruning False Positives in Static Analysis Warnings Through Function-Level Fuzzing

Authors: Aniruddhan Murali, Noble Saji Mathews, Mahmoud Alfadel, Meiyappan Nagappan, Meng Xu

Abstract: Manual confirmation of static analysis reports is a daunting task. This is due to both the large number of warnings and the high density of false positives among them. Fuzzing techniques have been proposed to verify static analysis warnings. However, a major limitation is that fuzzing the whole project to reach all static analysis warnings is not feasible. This can take several days and exponentia… ▽ More Manual confirmation of static analysis reports is a daunting task. This is due to both the large number of warnings and the high density of false positives among them. Fuzzing techniques have been proposed to verify static analysis warnings. However, a major limitation is that fuzzing the whole project to reach all static analysis warnings is not feasible. This can take several days and exponential machine time to increase code coverage linearly. Therefore, we propose FuzzSlice, a novel framework that automatically prunes possible false positives among static analysis warnings. Unlike prior work that mostly focuses on confirming true positives among static analysis warnings, which requires end-to-end fuzzing, FuzzSlice focuses on ruling out potential false positives, which are the majority in static analysis reports. The key insight that we base our work on is that a warning that does not yield a crash when fuzzed at the function level in a given time budget is a possible false positive. To achieve this, FuzzSlice first aims to generate compilable code slices at the function level and then fuzzes these code slices instead of the entire binary. FuzzSlice is also unlikely to misclassify a true bug as a false positive because the crashing input can be reproduced by a fuzzer at the function level as well. We evaluate FuzzSlice on the Juliet synthetic dataset and real-world complex C projects. Our evaluation shows that the ground truth in the Juliet dataset had 864 false positives which were all detected by FuzzSlice. For the open-source repositories, we were able to get the developers from two of these open-source repositories to independently label these warnings. FuzzSlice automatically identifies 33 out of 53 false positives confirmed by developers in these two repositories. Thus FuzzSlice reduces false positives by 62.26% in the open-source repositories and by 100% in the Juliet dataset. △ Less

Submitted 2 February, 2024; originally announced February 2024.

Comments: The paper has been accepted for publication at ICSE 2024 (Research Track)

LLbezpeky: Leveraging Large Language Models for Vulnerability Detection

Authors: Noble Saji Mathews, Yelizaveta Brus, Yousra Aafer, Meiyappan Nagappan, Shane McIntosh

Abstract: Despite the continued research and progress in building secure systems, Android applications continue to be ridden with vulnerabilities, necessitating effective detection methods. Current strategies involving static and dynamic analysis tools come with limitations like overwhelming number of false positives and limited scope of analysis which make either difficult to adopt. Over the past years, ma… ▽ More Despite the continued research and progress in building secure systems, Android applications continue to be ridden with vulnerabilities, necessitating effective detection methods. Current strategies involving static and dynamic analysis tools come with limitations like overwhelming number of false positives and limited scope of analysis which make either difficult to adopt. Over the past years, machine learning based approaches have been extensively explored for vulnerability detection, but its real-world applicability is constrained by data requirements and feature engineering challenges. Large Language Models (LLMs), with their vast parameters, have shown tremendous potential in understanding semnatics in human as well as programming languages. We dive into the efficacy of LLMs for detecting vulnerabilities in the context of Android security. We focus on building an AI-driven workflow to assist developers in identifying and rectifying vulnerabilities. Our experiments show that LLMs outperform our expectations in finding issues within applications correctly flagging insecure apps in 91.67% of cases in the Ghera benchmark. We use inferences from our experiments towards building a robust and actionable vulnerability detection system and demonstrate its effectiveness. Our experiments also shed light on how different various simple configurations can affect the True Positive (TP) and False Positive (FP) rates. △ Less

Submitted 13 February, 2024; v1 submitted 2 January, 2024; originally announced January 2024.

Comments: This project report was presented as a part of the course CS858 at the University of Waterloo under the supervision of Prof. Yousra Aafer

COMEX: A Tool for Generating Customized Source Code Representations

Authors: Debeshee Das, Noble Saji Mathews, Alex Mathai, Srikanth Tamilselvam, Kranthi Sedamaki, Sridhar Chimalakonda, Atul Kumar

Abstract: Learning effective representations of source code is critical for any Machine Learning for Software Engineering (ML4SE) system. Inspired by natural language processing, large language models (LLMs) like Codex and CodeGen treat code as generic sequences of text and are trained on huge corpora of code data, achieving state of the art performance on several software engineering (SE) tasks. However, v… ▽ More Learning effective representations of source code is critical for any Machine Learning for Software Engineering (ML4SE) system. Inspired by natural language processing, large language models (LLMs) like Codex and CodeGen treat code as generic sequences of text and are trained on huge corpora of code data, achieving state of the art performance on several software engineering (SE) tasks. However, valid source code, unlike natural language, follows a strict structure and pattern governed by the underlying grammar of the programming language. Current LLMs do not exploit this property of the source code as they treat code like a sequence of tokens and overlook key structural and semantic properties of code that can be extracted from code-views like the Control Flow Graph (CFG), Data Flow Graph (DFG), Abstract Syntax Tree (AST), etc. Unfortunately, the process of generating and integrating code-views for every programming language is cumbersome and time consuming. To overcome this barrier, we propose our tool COMEX - a framework that allows researchers and developers to create and combine multiple code-views which can be used by machine learning (ML) models for various SE tasks. Some salient features of our tool are: (i) it works directly on source code (which need not be compilable), (ii) it currently supports Java and C#, (iii) it can analyze both method-level snippets and program-level snippets by using both intra-procedural and inter-procedural analysis, and (iv) it is easily extendable to other languages as it is built on tree-sitter - a widely used incremental parser that supports over 40 languages. We believe this easy-to-use code-view generation and customization tool will give impetus to research in source code representation learning methods and ML4SE. Tool: https://pypi.org/project/comex - GitHub: https://github.com/IBM/tree-sitter-codeviews - Demo: https://youtu.be/GER6U87FVbU △ Less

Submitted 10 July, 2023; originally announced July 2023.

Comments: The paper has been accepted for publication at ASE 2023 (Tool Demonstrations Track)

NoteG: A Computational Notebook to Facilitate Rapid Game Prototyping

Authors: Noble Saji Mathews, Sridhar Chimalakonda

Abstract: Game development-based approaches are increasingly used to design curricula that can engage students, as these can help them apply and practice learnt computer science concepts. However, it can become complex to develop a minimum working game or a prototype with the help of high-end game engines. Game prototyping is one of the most essential parts of the game design and development cycle as it all… ▽ More Game development-based approaches are increasingly used to design curricula that can engage students, as these can help them apply and practice learnt computer science concepts. However, it can become complex to develop a minimum working game or a prototype with the help of high-end game engines. Game prototyping is one of the most essential parts of the game design and development cycle as it allows developers to continuously test and improve their ideas. In recent years, computational notebooks have gained widespread popularity among developers. They can help run individual code snippets, visualize the output, consolidate the source code, and share live code easily. However, its use has not been explored in the field of game development and prototyping. In this paper, we propose NoteG, a computational notebook towards rapid game prototyping. We evaluated the tool with 18 novice game developers through a questionnaire-based user survey. A majority of the volunteers (66%) found it easy to use and were of the opinion that it saves time. A few of the participants successfully extended the existing framework to implement new game mechanics within their prototypes. △ Less

Submitted 20 June, 2022; originally announced June 2022.

VedicViz: Towards Visualizing Vedic Principles in Mental Arithmetic

Authors: Noble Saji Mathews, Akhila Sri Manasa Venigalla, Sridhar Chimalakonda

Abstract: Augmenting teaching with visualization can help students understand concepts better. Researchers have leveraged visualization to teach conventional mathematics some examples being spatial and origami visualizations. Apart from conventional mathematics, systems such as mental arithmetic involve techniques for rapid calculation without the use of any computing tools and hence have been used in devel… ▽ More Augmenting teaching with visualization can help students understand concepts better. Researchers have leveraged visualization to teach conventional mathematics some examples being spatial and origami visualizations. Apart from conventional mathematics, systems such as mental arithmetic involve techniques for rapid calculation without the use of any computing tools and hence have been used in developing computational competence among students. Vedic Mathematics is one such set of techniques for mental computation. However, there is a lack of technical tools which tackle mental arithmetic concepts and provide aid in the teaching of these topics to school students. Therefore, we propose VedicViz, a web portal that provides dynamic visualization of mathematical operations such as addition, multiplication and square root calculation, based on techniques in Vedic Mathematics. The web portal also provides visualization that enables learners to compare and contrast the mental mathematics based approach with the traditional methods for various inputs and operations. We evaluated VedicViz with 20 volunteers, who were in their high school education level. They found our web portal to be useful in practicing and learning to use the methods to perform various mathematical operations. △ Less

Submitted 18 May, 2022; originally announced May 2022.

Comments: 13 pages, 5 figures, 2 tables

On the Impact of Multiple Source Code Representations on Software Engineering Tasks -- An Empirical Study

Authors: Karthik Chandra Swarna, Noble Saji Mathews, Dheeraj Vagavolu, Sridhar Chimalakonda

Abstract: Efficiently representing source code is crucial for various software engineering tasks such as code classification and clone detection. Existing approaches primarily use Abstract Syntax Tree (AST), and only a few focus on semantic graphs such as Control Flow Graph (CFG) and Program Dependency Graph (PDG), which contain information about source code that AST does not. Even though some works tried t… ▽ More Efficiently representing source code is crucial for various software engineering tasks such as code classification and clone detection. Existing approaches primarily use Abstract Syntax Tree (AST), and only a few focus on semantic graphs such as Control Flow Graph (CFG) and Program Dependency Graph (PDG), which contain information about source code that AST does not. Even though some works tried to utilize multiple representations, they do not provide any insights about the costs and benefits of using multiple representations. The primary goal of this paper is to discuss the implications of utilizing multiple code representations, specifically AST, CFG, and PDG. We modify an AST path-based approach to accept multiple representations as input to an attention-based model. We do this to measure the impact of additional representations (such as CFG and PDG) over AST. We evaluate our approach on three tasks: Method Naming, Program Classification, and Clone Detection. Our approach increases the performance on these tasks by 11% (F1), 15.7% (Accuracy), and 9.3% (F1), respectively, over the baseline. In addition to the effect on performance, we discuss timing overheads incurred with multiple representations. We envision this work providing researchers with a lens to evaluate combinations of code representations for various tasks. △ Less

Submitted 24 December, 2023; v1 submitted 21 June, 2021; originally announced June 2021.

Detox Browser -- Towards Filtering Sensitive Content On the Web

Authors: Noble Saji Mathews, Sridhar Chimalakonda

Abstract: The annual consumption of web-based resources is increasing at a very fast rate, mainly due to an increase in affordability and accessibility of the internet. Many are relying on the web to get diverse perspectives, but at the same time, it can expose them to content that is harmful to their mental well-being. Catchy headlines and emotionally charged articles increase the number of readers which i… ▽ More The annual consumption of web-based resources is increasing at a very fast rate, mainly due to an increase in affordability and accessibility of the internet. Many are relying on the web to get diverse perspectives, but at the same time, it can expose them to content that is harmful to their mental well-being. Catchy headlines and emotionally charged articles increase the number of readers which in turn increases ad revenue for websites. When a user consumes a large quantity of negative content, it adversely impacts the user's happiness and has a significant impact on his/her mood and state of mind. Many studies carried out during the COVID-19 pandemic has shown that people across the globe irrespective of their country of origin have experienced higher levels of anxiety and depression. Web filters can help in constructing a digital environment that is more suitable for people prone to depression, anxiety and stress. A significant amount of work has been done in the field of web filtering, but there has been limited focus on helping Highly Sensitive Persons (HSP's) or those with stress disorders induced by trauma. Through this paper, we propose detox Browser, a simple tool that enables end-users to tune out of or control their exposure to topics that can affect their mental well being. The extension makes use of sentiment analysis and keywords to filter out flagged content from google search results and warns users if any blacklisted topics are detected when navigating across websites △ Less

Submitted 18 June, 2021; originally announced June 2021.

Comments: 6 pages, 2 figures, CSCW

AiR -- An Augmented Reality Application for Visualizing Air Pollution

Authors: Noble Saji Mathews, Sridhar Chimalakonda, Suresh Jain

Abstract: Air quality is a term used to describe the concentration levels of various pollutants in the air we breathe. The air quality, which is degrading rapidly across the globe, has been a source of great concern. Across the globe, governments are taking various measures to reduce air pollution. Bringing awareness about environmental pollution among the public plays a major role in controlling air pollut… ▽ More Air quality is a term used to describe the concentration levels of various pollutants in the air we breathe. The air quality, which is degrading rapidly across the globe, has been a source of great concern. Across the globe, governments are taking various measures to reduce air pollution. Bringing awareness about environmental pollution among the public plays a major role in controlling air pollution, as the programs proposed by governments require the support of the public. Though information on air quality is present on multiple portals such as the Central Pollution Control Board (CPCB), which provides Air Quality Index that could be accessed by the public. However, such portals are scarcely visited by the general public. Visualizing air quality in the location where an individual resides could help in bringing awareness among the public. This visualization could be rendered using Augmented Reality techniques. Considering the widespread usage of Android based mobile devices in India, and the importance of air quality visualization, we present AiR, as an Android based mobile application. AiR considers the air quality measured by CPCB, in a locality that is detected by the user's GPS or in a locality of user's choice, and visualizes various air pollutants present in the locality $(PM_1{}_0, PM_2{}_.{}_5, NO_2, SO_2, CO, O_3 \& NH_3)$ and displays them in the user's surroundings. AiR also creates awareness in an interactive manner about the different pollutants, sources, and their impacts on health. △ Less

Submitted 3 June, 2020; originally announced June 2020.

Comments: 18 pages and 16 figures