-
On Applying Bandit Algorithm to Fault Localization Techniques
Authors:
Masato Nakao,
Kensei Hamamoto,
Masateru Tsunoda,
Amjed Tahir,
Koji Toda,
Akito Monden,
Keitaro Nakasai,
Kenichi Matsumoto
Abstract:
Developers must select a high-performance fault localization (FL) technique from available ones. A conventional approach is to try to select only one FL technique that is expected to attain high performance before debugging activity. In contrast, we propose a new approach that dynamically selects better FL techniques during debugging activity.
Developers must select a high-performance fault localization (FL) technique from available ones. A conventional approach is to try to select only one FL technique that is expected to attain high performance before debugging activity. In contrast, we propose a new approach that dynamically selects better FL techniques during debugging activity.
△ Less
Submitted 10 September, 2024;
originally announced September 2024.
-
An Empirical Study of the Impact of Test Strategies on Online Optimization for Ensemble-Learning Defect Prediction
Authors:
Kensei Hamamoto,
Masateru Tsunoda,
Amjed Tahir,
Kwabena Ebo Bennin,
Akito Monden,
Koji Toda,
Keitaro Nakasai,
Kenichi Matsumoto
Abstract:
Ensemble learning methods have been used to enhance the reliability of defect prediction models. However, there is an inconclusive stability of a single method attaining the highest accuracy among various software projects. This work aims to improve the performance of ensemble-learning defect prediction among such projects by helping select the highest accuracy ensemble methods. We employ bandit a…
▽ More
Ensemble learning methods have been used to enhance the reliability of defect prediction models. However, there is an inconclusive stability of a single method attaining the highest accuracy among various software projects. This work aims to improve the performance of ensemble-learning defect prediction among such projects by helping select the highest accuracy ensemble methods. We employ bandit algorithms (BA), an online optimization method, to select the highest-accuracy ensemble method. Each software module is tested sequentially, and bandit algorithms utilize the test outcomes of the modules to evaluate the performance of the ensemble learning methods. The test strategy followed might impact the testing effort and prediction accuracy when applying online optimization. Hence, we analyzed the test order's influence on BA's performance. In our experiment, we used six popular defect prediction datasets, four ensemble learning methods such as bagging, and three test strategies such as testing positive-prediction modules first (PF). Our results show that when BA is applied with PF, the prediction accuracy improved on average, and the number of found defects increased by 7% on a minimum of five out of six datasets (although with a slight increase in the testing effort by about 4% from ordinal ensemble learning). Hence, BA with PF strategy is the most effective to attain the highest prediction accuracy using ensemble methods on various projects.
△ Less
Submitted 10 September, 2024;
originally announced September 2024.
-
The Impact of Defect (Re) Prediction on Software Testing
Authors:
Yukasa Murakami,
Yuta Yamasaki,
Masateru Tsunoda,
Akito Monden,
Amjed Tahir,
Kwabena Ebo Bennin,
Koji Toda,
Keitaro Nakasai
Abstract:
Cross-project defect prediction (CPDP) aims to use data from external projects as historical data may not be available from the same project. In CPDP, deciding on a particular historical project to build a training model can be difficult. To help with this decision, a Bandit Algorithm (BA) based approach has been proposed in prior research to select the most suitable learning project. However, thi…
▽ More
Cross-project defect prediction (CPDP) aims to use data from external projects as historical data may not be available from the same project. In CPDP, deciding on a particular historical project to build a training model can be difficult. To help with this decision, a Bandit Algorithm (BA) based approach has been proposed in prior research to select the most suitable learning project. However, this BA method could lead to the selection of unsuitable data during the early iteration of BA (i.e., early stage of software testing). Selecting an unsuitable model can reduce the prediction accuracy, leading to potential defect overlooking. This study aims to improve the BA method to reduce defects overlooking, especially during the early testing stages. Once all modules have been tested, modules tested in the early stage are re-predicted, and some modules are retested based on the re-prediction. To assess the impact of re-prediction and retesting, we applied five kinds of BA methods, using 8, 16, and 32 OSS projects as learning data. The results show that the newly proposed approach steadily reduced the probability of defect overlooking without degradation of prediction accuracy.
△ Less
Submitted 10 September, 2024; v1 submitted 16 April, 2024;
originally announced April 2024.
-
Building Defect Prediction Models by Online Learning Considering Defect Overlooking
Authors:
Nikolay Fedorov,
Yuta Yamasaki,
Masateru Tsunoda,
Akito Monden,
Amjed Tahir,
Kwabena Ebo Bennin,
Koji Toda,
Keitaro Nakasai
Abstract:
Building defect prediction models based on online learning can enhance prediction accuracy. It continuously rebuilds a new prediction model, when a new data point is added. However, a module predicted as "non-defective" can result in fewer test cases for such modules. Thus, a defective module can be overlooked during testing. The erroneous test results are used as learning data by online learning,…
▽ More
Building defect prediction models based on online learning can enhance prediction accuracy. It continuously rebuilds a new prediction model, when a new data point is added. However, a module predicted as "non-defective" can result in fewer test cases for such modules. Thus, a defective module can be overlooked during testing. The erroneous test results are used as learning data by online learning, which could negatively affect prediction accuracy. To suppress the negative influence, we propose to apply a method that fixes the prediction as positive during the initial stage of online learning. Additionally, we improved the method to consider the probability of the overlooking. In our experiment, we demonstrate this negative influence on prediction accuracy, and the effectiveness of our approach. The results show that our approach did not negatively affect AUC but significantly improved recall.
△ Less
Submitted 16 April, 2024;
originally announced April 2024.
-
Selecting Source Code Generation Tools Based on Bandit Algorithms
Authors:
Ryoto Shima,
Masateru Tsunoda,
Yukasa Murakami,
Akito Monden,
Amjed Tahir,
Kwabena Ebo Bennin,
Koji Toda,
Keitaro Nakasai
Abstract:
Background: Recently, code generation tools such as ChatGPT have drawn attention to their performance. Generally, a prior analysis of their performance is needed to select new code-generation tools from a list of candidates. Without such analysis, there is a higher risk of selecting an ineffective tool, negatively affecting software development productivity. Additionally, conducting prior analysis…
▽ More
Background: Recently, code generation tools such as ChatGPT have drawn attention to their performance. Generally, a prior analysis of their performance is needed to select new code-generation tools from a list of candidates. Without such analysis, there is a higher risk of selecting an ineffective tool, negatively affecting software development productivity. Additionally, conducting prior analysis of new code generation tools takes time and effort. Aim: To use a new code generation tool without prior analysis but with low risk, we propose to evaluate the new tools during software development (i.e., online optimization). Method: We apply the bandit algorithm (BA) approach to help select the best code-generation tool among candidates. Developers evaluate whether the result of the tool is correct or not. When code generation and evaluation are repeated, the evaluation results are saved. We utilize the stored evaluation results to select the best tool based on the BA approach. Our preliminary analysis evaluated five code generation tools with 164 code generation cases using BA. Result: The BA approach selected ChatGPT as the best tool as the evaluation proceeded, and during the evaluation, the average accuracy by the BA approach outperformed the second-best performing tool. Our results reveal the feasibility and effectiveness of BA in assisting the selection of best-performing code generation tools.
△ Less
Submitted 20 December, 2023;
originally announced December 2023.
-
Software Defect Prediction by Online Learning Considering Defect Overlooking
Authors:
Yuta Yamasaki,
Nikolay Fedorov,
Masateru Tsunoda,
Akito Monden,
Amjed Tahir,
Kwabena Ebo Bennin,
Koji Toda,
Keitaro Nakasai
Abstract:
Building defect prediction models based on online learning can enhance prediction accuracy. It continuously rebuilds a new prediction model when adding a new data point. However, predicting a module as "non-defective" (i.e., negative prediction) can result in fewer test cases for such modules. Therefore, defects can be overlooked during testing, even when the module is defective. The erroneous tes…
▽ More
Building defect prediction models based on online learning can enhance prediction accuracy. It continuously rebuilds a new prediction model when adding a new data point. However, predicting a module as "non-defective" (i.e., negative prediction) can result in fewer test cases for such modules. Therefore, defects can be overlooked during testing, even when the module is defective. The erroneous test results are used as learning data by online learning, which could negatively affect prediction accuracy. In our experiment, we demonstrate this negative influence on prediction accuracy.
△ Less
Submitted 25 August, 2023;
originally announced August 2023.
-
Analyzing and Calibrating Risk Assessment by Software Developers
Authors:
Yukasa Murakami,
Masateru Tsunoda,
Eduardo C. Campos
Abstract:
In software project management, risk management is a critical factor. Project managers use existing lists of risk or perform brainstorming to identify the risks. However, it is not easy to perceive all the risks objectively. As a result, some risks are perceived based on subjective impression, which leads to risk biases. So, our goals are (i) We clarify the risk perception of developers to enhance…
▽ More
In software project management, risk management is a critical factor. Project managers use existing lists of risk or perform brainstorming to identify the risks. However, it is not easy to perceive all the risks objectively. As a result, some risks are perceived based on subjective impression, which leads to risk biases. So, our goals are (i) We clarify the risk perception of developers to enhance the reliability of the brainstorming, and (ii) we calibrate the risk assessment based on a mathematical model to make more accurate risk list. In the analysis, we collected data concerning the risk perception of 69 professional software developers via a questionnaire. The average number of years of experience among these professionals was 18.3. Using the dataset, we applied factor analysis to clarify the factors that affect the evaluation of risk impact. The questionnaire was based on the risk perception theory established by Slovic, in which "dread" and "unknown" are the major factor of risk perception. The analysis result shows that (i) risk experience (i.e., whether a developer actually faced the risk or not) sometimes affects risk assessment (evaluation of risk impact), (ii) risk perception is considered to be based on unknown and dread factors, and (iii) risk assessment can be calibrated by a mathematical model (the average absolute error was 0.20).
△ Less
Submitted 7 September, 2022;
originally announced September 2022.
-
Relationship between Gender and Code Reading Speed in Software Development
Authors:
Yuriko Takatsuka,
Yukasa Murakami,
Masateru Tsunoda,
Masahide Nakamura
Abstract:
Recently, workforce shortage has become a popular issue in information technology (IT). One solution to increasing the workforce supply is to increase the number of female IT professionals. This is because there is gender imbalance in information technology area. To accomplish this, it is important to suppress the influence of biases, such as the belief that men are more suited for careers in scie…
▽ More
Recently, workforce shortage has become a popular issue in information technology (IT). One solution to increasing the workforce supply is to increase the number of female IT professionals. This is because there is gender imbalance in information technology area. To accomplish this, it is important to suppress the influence of biases, such as the belief that men are more suited for careers in science and technology than women, and to increase the choice of careers available to female professionals. To help suppress the influence of gender bias, we analyzed the relationship between gender and code reading speed in the field of software development. Certain source codes require developers to use substantial memory to properly understand them, such as those with many variables that frequently change values. Several studies have indicated that the performance of memory differs in males and females. To test the veracity of this claim, we analyzed the influence of gender on code-reading speed through an experiment. Pursuant to this, we prepared four programs that required varied amounts of memory to properly understand them. Then, we measured the time required by each of the 17 male and 16 female subjects (33 subjects in total) to comprehend the different programs. The results suggest that there is no explicit difference between male and female subjects in this regard, even in the case of programs that require high memory capacities for proper understanding.
△ Less
Submitted 7 September, 2022;
originally announced September 2022.
-
Measuring Cognitive Load of Software Developers Based on Nasal Skin Temperature
Authors:
Keitaro Nakasai,
Shin Komeda,
Masateru Tsunoda,
Masayuki Kashima
Abstract:
It has recently become increasingly important to measure the cognitive load of developers. This is because continuing with a development under a high cognitive load may cause human errors. Therefore, to automatically measure the cognitive load, existing studies have used biometric measures such as brain waves and the heart rate. However, developers are often required to equip certain devices when…
▽ More
It has recently become increasingly important to measure the cognitive load of developers. This is because continuing with a development under a high cognitive load may cause human errors. Therefore, to automatically measure the cognitive load, existing studies have used biometric measures such as brain waves and the heart rate. However, developers are often required to equip certain devices when measuring them, and can therefore be physically burdened. In this study, we evaluated the feasibility of non-invasive biometric measures based on the nasal skin temperature. The nasal skin temperature has been widely used in other fields to measure mental status. In the present experiment, the subjects created small Java programs, and we estimated their cognitive load using the proposed biometric measures based on the nasal skin temperature. As a result, the proposed biometric measures were shown to be more effective than non-biometric measures. Hence, biometric measures based on nasal skin temperature are promising for estimating the cognitive load of developers.
△ Less
Submitted 1 April, 2022;
originally announced April 2022.
-
A Simulation Study of Bandit Algorithms to Address External Validity of Software Fault Prediction
Authors:
Teruki Hayakawa,
Masateru Tsunoda,
Koji Toda,
Keitaro Nakasai,
Kenichi Matsumoto
Abstract:
Various software fault prediction models and techniques for building algorithms have been proposed. Many studies have compared and evaluated them to identify the most effective ones. However, in most cases, such models and techniques do not have the best performance on every dataset. This is because there is diversity of software development datasets, and therefore, there is a risk that the select…
▽ More
Various software fault prediction models and techniques for building algorithms have been proposed. Many studies have compared and evaluated them to identify the most effective ones. However, in most cases, such models and techniques do not have the best performance on every dataset. This is because there is diversity of software development datasets, and therefore, there is a risk that the selected model or technique shows bad performance on a certain dataset. To avoid selecting a low accuracy model, we apply bandit algorithms to predict faults. Consider a case where player has 100 coins to bet on several slot machines. Ordinary usage of software fault prediction is analogous to the player betting all 100 coins in one slot machine. In contrast, bandit algorithms bet one coin on each machine (i.e., use prediction models) step-by-step to seek the best machine. In the experiment, we developed an artificial dataset that includes 100 modules, 15 of which include faults. Then, we developed various artificial fault prediction models and selected them dynamically using bandit algorithms. The Thomson sampling algorithm showed the best or second-best prediction performance compared with using only one prediction model.
△ Less
Submitted 17 March, 2020; v1 submitted 10 March, 2020;
originally announced March 2020.
