Showing 1–50 of 99 results for author: Thomas, J

An Open-Source American Sign Language Fingerspell Recognition and Semantic Pose Retrieval Interface

Authors: Kevin Jose Thomas

Abstract: This paper introduces an open-source interface for American Sign Language fingerspell recognition and semantic pose retrieval, aimed to serve as a stepping stone towards more advanced sign language translation systems. Utilizing a combination of convolutional neural networks and pose estimation models, the interface provides two modular components: a recognition module for translating ASL fingersp… ▽ More This paper introduces an open-source interface for American Sign Language fingerspell recognition and semantic pose retrieval, aimed to serve as a stepping stone towards more advanced sign language translation systems. Utilizing a combination of convolutional neural networks and pose estimation models, the interface provides two modular components: a recognition module for translating ASL fingerspelling into spoken English and a production module for converting spoken English into ASL pose sequences. The system is designed to be highly accessible, user-friendly, and capable of functioning in real-time under varying environmental conditions like backgrounds, lighting, skin tones, and hand sizes. We discuss the technical details of the model architecture, application in the wild, as well as potential future enhancements for real-world consumer applications. △ Less

Submitted 17 August, 2024; originally announced August 2024.

Comments: 8 pages, 9 figures

Quantum Circuit Optimization: Current trends and future direction

Authors: Geetha Karuppasamy, Varun Puram, Stevens Johnson, Johnson P Thomas

Abstract: Optimization of quantum circuits for a given problem is very important in order to achieve faster calculations as well as reduce errors due to noise. Optimization has to be achieved while ensuring correctness at all times. In this survey paper, recent advancements in quantum circuit optimization are explored. Both hardware independent as well as hardware dependent optimization are presented. State… ▽ More Optimization of quantum circuits for a given problem is very important in order to achieve faster calculations as well as reduce errors due to noise. Optimization has to be achieved while ensuring correctness at all times. In this survey paper, recent advancements in quantum circuit optimization are explored. Both hardware independent as well as hardware dependent optimization are presented. State-of-the-art methods for optimizing quantum circuits, including analytical algorithms, heuristic algorithms, machine learning-based algorithms, and hybrid quantum-classical algorithms are discussed. Additionally, the advantages and disadvantages of each method and the challenges associated with them are highlighted. Moreover, the potential research opportunities in this field are also discussed. △ Less

Submitted 16 August, 2024; originally announced August 2024.

Quantum Algorithm for Jaccard Similarity

Authors: Varun Puram, Ruthvik Rao Bobbili, Johnson P Thomas

Abstract: Jaccard Similarity is a very common proximity measurement used to compute the similarity between two asymmetric binary vectors. Jaccard Similarity is the ratio between the 1s (Intersection of two vectors) to 1s (Union of two vectors). This paper introduces a quantum algorithm for finding the Jaccard Similarity 1s, in the Intersection and Union of two binary vectors. There are two sub-algorithms on… ▽ More Jaccard Similarity is a very common proximity measurement used to compute the similarity between two asymmetric binary vectors. Jaccard Similarity is the ratio between the 1s (Intersection of two vectors) to 1s (Union of two vectors). This paper introduces a quantum algorithm for finding the Jaccard Similarity 1s, in the Intersection and Union of two binary vectors. There are two sub-algorithms one for each. Measuring the register for respective algorithm gives count of number of 1 s in binary format. Implementation on IBM composer is also included. △ Less

Submitted 16 August, 2024; originally announced August 2024.

Methods to Measure the Broncho-Arterial Ratio and Wall Thickness in the Right Lower Lobe for Defining Radiographic Reversibility of Bronchiectasis

Authors: Abhijith R. Beeravolu, Ian Brent Masters, Mirjam Jonkman, Kheng Cher Yeo, Spyridon Prountzos, Rahul J Thomas, Eva Ignatious, Sami Azam, Gabrielle B McCallum, Efthymia Alexopoulou, Anne B Chang, Friso De Boer

Abstract: The diagnosis of bronchiectasis requires measuring abnormal bronchial dilation. It is confirmed using a chest CT scan, where the key feature is an increased broncho-arterial ratio (BAR) (>0.8 in children), often with bronchial wall thickening. Image processing methods facilitate quicker interpretation and detailed evaluations by lobes and segments. Challenges like inclined nature, oblique orientat… ▽ More The diagnosis of bronchiectasis requires measuring abnormal bronchial dilation. It is confirmed using a chest CT scan, where the key feature is an increased broncho-arterial ratio (BAR) (>0.8 in children), often with bronchial wall thickening. Image processing methods facilitate quicker interpretation and detailed evaluations by lobes and segments. Challenges like inclined nature, oblique orientation, and partial volume effect make it difficult to obtain accurate measurements in the upper and middle lobes using the same algorithms. Therefore, accurate detection and measurement of airway and artery regions for BAR and wall thickness in each lobe require different image processing/machine learning methods. We propose methods for: 1. Separating the right lower lobe (RLL) region from full-length CT scans using the tracheal bifurcation (Carina) point as a central marker; 2. Locating the inner diameter of airways and outer diameter of arteries for BAR measurement; and 3. Measuring airway wall thickness (WT) by identifying the outer and inner diameters of airway boundaries. Analysis of 13 HRCT scans with varying thicknesses (0.67mm, 1mm, 2mm) shows the tracheal bifurcation frame can be detected accurately, with a deviation of +/- 2 frames in some cases. A Windows app was developed for measuring inner airway diameter, artery diameter, BAR, and wall thickness, allowing users to draw boundaries around visible BA pairs in the RLL region. Measurements of 10 BA pairs revealed accurate results comparable to those of a human reader, with deviations of +/- 0.10-0.15mm. Additional studies and validation are needed to consolidate inter- and intra-rater variability and enhance the methods. △ Less

Submitted 18 July, 2024; originally announced July 2024.

Comments: 14 pages

Optimizing Nurse Scheduling: A Supply Chain Approach for Healthcare Institutions

Authors: Jubin Thomas

Abstract: When managing an organization, planners often encounter numerous challenging scenarios. In such instances, relying solely on intuition or managerial experience may not suffice, necessitating a quantitative approach. This demand is further accentuated in the era of big data, where the sheer scale and complexity of constraints pose significant challenges. Therefore, the aim of this study is to provi… ▽ More When managing an organization, planners often encounter numerous challenging scenarios. In such instances, relying solely on intuition or managerial experience may not suffice, necessitating a quantitative approach. This demand is further accentuated in the era of big data, where the sheer scale and complexity of constraints pose significant challenges. Therefore, the aim of this study is to provide a foundational framework for addressing personnel scheduling, a critical issue in organizational management. Specifically, we focus on optimizing shift assignments for staff, a task fraught with complexities due to factors such as contractual obligations and mandated rest periods. Moreover, the current landscape is characterized by frequent employee shortages across various industries, with many organizations lacking efficient and dependable management tools to address them. Therefore, our attention is particularly drawn to the nurse rostering problem, a personnel scheduling challenge prevalent in healthcare settings. These issues are characterized by a multitude of variables, given that a single healthcare facility may employ hundreds of nurses, alongside stringent constraints such as the need for adequate staffing levels and rest periods postnight shifts. Furthermore, the ongoing COVID19 pandemic has exacerbated staffing challenges in healthcare institutions, underlining the importance of accurately assessing staffing needs and optimizing shift allocations for effective operation amidst crisis situations. △ Less

Submitted 29 May, 2024; originally announced July 2024.

Journal ref: Vol. 20 No. 6s (2024)

Graph Reinforcement Learning for Power Grids: A Comprehensive Survey

Authors: Mohamed Hassouna, Clara Holzhüter, Pawel Lytaev, Josephine Thomas, Bernhard Sick, Christoph Scholz

Abstract: The rise of renewable energy and distributed generation requires new approaches to overcome the limitations of traditional methods. In this context, Graph Neural Networks are promising due to their ability to learn from graph-structured data. Combined with Reinforcement Learning, they can serve as control approaches to determine remedial network actions. This review analyses how Graph Reinforcemen… ▽ More The rise of renewable energy and distributed generation requires new approaches to overcome the limitations of traditional methods. In this context, Graph Neural Networks are promising due to their ability to learn from graph-structured data. Combined with Reinforcement Learning, they can serve as control approaches to determine remedial network actions. This review analyses how Graph Reinforcement Learning (GRL) can improve representation learning and decision making in power grid use cases. Although GRL has demonstrated adaptability to unpredictable events and noisy data, it is primarily at a proof-of-concept stage. We highlight open challenges and limitations with respect to real-world applications. △ Less

Submitted 26 August, 2024; v1 submitted 5 July, 2024; originally announced July 2024.

State-Based Automation for Time-Restricted Eating Adherence

Authors: Samuel E. Armstrong, Aaron D. Mullen, J. Matthew Thomas, Dorothy D. Sears, Julie S. Pendergast, Jeffrey Talbert, Cody Bumgardner

Abstract: Developing and enforcing study protocols is a foundational component of medical research. As study complexity for participant interactions increases, translating study protocols to supporting application code becomes challenging. A collaboration exists between the University of Kentucky and Arizona State University to determine the efficacy of time-restricted eating in improving metabolic risk amo… ▽ More Developing and enforcing study protocols is a foundational component of medical research. As study complexity for participant interactions increases, translating study protocols to supporting application code becomes challenging. A collaboration exists between the University of Kentucky and Arizona State University to determine the efficacy of time-restricted eating in improving metabolic risk among postmenopausal women. This study utilizes a graph-based approach to monitor and support adherence to a designated schedule, enabling the validation and step-wise audit of participants' statuses to derive dependable conclusions. A texting service, driven by a participant graph, automatically manages interactions and collects data. Participant data is then accessible to the research study team via a website, which enables viewing, management, and exportation. This paper presents a system for automatically managing participants in a time-restricted eating study that eliminates time-consuming interactions with participants. △ Less

Submitted 26 June, 2024; originally announced June 2024.

Comments: 8 pages, 4 figures, submitted to AMIA 2024 Annual Symposium

ASoBO: Attentive Beamformer Selection for Distant Speaker Diarization in Meetings

Authors: Theo Mariotte, Anthony Larcher, Silvio Montresor, Jean-Hugh Thomas

Abstract: Speaker Diarization (SD) aims at grouping speech segments that belong to the same speaker. This task is required in many speech-processing applications, such as rich meeting transcription. In this context, distant microphone arrays usually capture the audio signal. Beamforming, i.e., spatial filtering, is a common practice to process multi-microphone audio data. However, it often requires an expli… ▽ More Speaker Diarization (SD) aims at grouping speech segments that belong to the same speaker. This task is required in many speech-processing applications, such as rich meeting transcription. In this context, distant microphone arrays usually capture the audio signal. Beamforming, i.e., spatial filtering, is a common practice to process multi-microphone audio data. However, it often requires an explicit localization of the active source to steer the filter. This paper proposes a self-attention-based algorithm to select the output of a bank of fixed spatial filters. This method serves as a feature extractor for joint Voice Activity (VAD) and Overlapped Speech Detection (OSD). The speaker diarization is then inferred from the detected segments. The approach shows convincing distant VAD, OSD, and SD performance, e.g. 14.5% DER on the AISHELL-4 dataset. The analysis of the self-attention weights demonstrates their explainability, as they correlate with the speaker's angular locations. △ Less

Submitted 5 June, 2024; originally announced June 2024.

Comments: 5 pages, 2 figures, 2 tables, accepted at Interspeech 2024

Exploring the use of a Large Language Model for data extraction in systematic reviews: a rapid feasibility study

Authors: Lena Schmidt, Kaitlyn Hair, Sergio Graziozi, Fiona Campbell, Claudia Kapp, Alireza Khanteymoori, Dawn Craig, Mark Engelbert, James Thomas

Abstract: This paper describes a rapid feasibility study of using GPT-4, a large language model (LLM), to (semi)automate data extraction in systematic reviews. Despite the recent surge of interest in LLMs there is still a lack of understanding of how to design LLM-based automation tools and how to robustly evaluate their performance. During the 2023 Evidence Synthesis Hackathon we conducted two feasibility… ▽ More This paper describes a rapid feasibility study of using GPT-4, a large language model (LLM), to (semi)automate data extraction in systematic reviews. Despite the recent surge of interest in LLMs there is still a lack of understanding of how to design LLM-based automation tools and how to robustly evaluate their performance. During the 2023 Evidence Synthesis Hackathon we conducted two feasibility studies. Firstly, to automatically extract study characteristics from human clinical, animal, and social science domain studies. We used two studies from each category for prompt-development; and ten for evaluation. Secondly, we used the LLM to predict Participants, Interventions, Controls and Outcomes (PICOs) labelled within 100 abstracts in the EBM-NLP dataset. Overall, results indicated an accuracy of around 80%, with some variability between domains (82% for human clinical, 80% for animal, and 72% for studies of human social sciences). Causal inference methods and study design were the data extraction items with the most errors. In the PICO study, participants and intervention/control showed high accuracy (>80%), outcomes were more challenging. Evaluation was done manually; scoring methods such as BLEU and ROUGE showed limited value. We observed variability in the LLMs predictions and changes in response quality. This paper presents a template for future evaluations of LLMs in the context of data extraction for systematic review automation. Our results show that there might be value in using LLMs, for example as second or third reviewers. However, caution is advised when integrating models such as GPT-4 into tools. Further research on stability and reliability in practical settings is warranted for each type of data that is processed by the LLM. △ Less

Submitted 23 May, 2024; originally announced May 2024.

Comments: Conference proceedings, peer-reviewed and presented at the 3rd Workshop on Augmented Intelligence for Technology-Assisted Reviews Systems, Glasgow, 2024

Journal ref: Proceedings of the 3rd Workshop on Augmented Intelligence for Technology-Assisted Reviews Systems, 2024

PRISM: Patient Records Interpretation for Semantic Clinical Trial Matching using Large Language Models

Authors: Shashi Kant Gupta, Aditya Basu, Mauro Nievas, Jerrin Thomas, Nathan Wolfrath, Adhitya Ramamurthi, Bradley Taylor, Anai N. Kothari, Regina Schwind, Therica M. Miller, Sorena Nadaf-Rahrov, Yanshan Wang, Hrituraj Singh

Abstract: Clinical trial matching is the task of identifying trials for which patients may be potentially eligible. Typically, this task is labor-intensive and requires detailed verification of patient electronic health records (EHRs) against the stringent inclusion and exclusion criteria of clinical trials. This process is manual, time-intensive, and challenging to scale up, resulting in many patients miss… ▽ More Clinical trial matching is the task of identifying trials for which patients may be potentially eligible. Typically, this task is labor-intensive and requires detailed verification of patient electronic health records (EHRs) against the stringent inclusion and exclusion criteria of clinical trials. This process is manual, time-intensive, and challenging to scale up, resulting in many patients missing out on potential therapeutic options. Recent advancements in Large Language Models (LLMs) have made automating patient-trial matching possible, as shown in multiple concurrent research studies. However, the current approaches are confined to constrained, often synthetic datasets that do not adequately mirror the complexities encountered in real-world medical data. In this study, we present the first, end-to-end large-scale empirical evaluation of clinical trial matching using real-world EHRs. Our study showcases the capability of LLMs to accurately match patients with appropriate clinical trials. We perform experiments with proprietary LLMs, including GPT-4 and GPT-3.5, as well as our custom fine-tuned model called OncoLLM and show that OncoLLM, despite its significantly smaller size, not only outperforms GPT-3.5 but also matches the performance of qualified medical doctors. All experiments were carried out on real-world EHRs that include clinical notes and available clinical trials from a single cancer center in the United States. △ Less

Submitted 26 April, 2024; v1 submitted 23 April, 2024; originally announced April 2024.

Comments: 30 Pages, 8 Figures, Supplementary Work Attached

Channel-Combination Algorithms for Robust Distant Voice Activity and Overlapped Speech Detection

Authors: Théo Mariotte, Anthony Larcher, Silvio Montrésor, Jean-Hugh Thomas

Abstract: Voice Activity Detection (VAD) and Overlapped Speech Detection (OSD) are key pre-processing tasks for speaker diarization. In the meeting context, it is often easier to capture speech with a distant device. This consideration however leads to severe performance degradation. We study a unified supervised learning framework to solve distant multi-microphone joint VAD and OSD (VAD+OSD). This paper in… ▽ More Voice Activity Detection (VAD) and Overlapped Speech Detection (OSD) are key pre-processing tasks for speaker diarization. In the meeting context, it is often easier to capture speech with a distant device. This consideration however leads to severe performance degradation. We study a unified supervised learning framework to solve distant multi-microphone joint VAD and OSD (VAD+OSD). This paper investigates various multi-channel VAD+OSD front-ends that weight and combine incoming channels. We propose three algorithms based on the Self-Attention Channel Combinator (SACC), previously proposed in the literature. Experiments conducted on the AMI meeting corpus exhibit that channel combination approaches bring significant VAD+OSD improvements in the distant speech scenario. Specifically, we explore the use of learned complex combination weights and demonstrate the benefits of such an approach in terms of explainability. Channel combination-based VAD+OSD systems are evaluated on the final back-end task, i.e. speaker diarization, and show significant improvements. Finally, since multi-channel systems are trained given a fixed array configuration, they may fail in generalizing to other array set-ups, e.g. mismatched number of microphones. A channel-number invariant loss is proposed to learn a unique feature representation regardless of the number of available microphones. The evaluation conducted on mismatched array configurations highlights the robustness of this training strategy. △ Less

Submitted 13 February, 2024; originally announced February 2024.

Comments: 14 pages, 5 figures, accepted at IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP)

The Devil Is in the Command Line: Associating the Compiler Flags With the Binary and Build Metadata

Authors: Gunnar Kudrjavets, Aditya Kumar, Jeff Thomas, Ayushi Rastogi

Abstract: Engineers build large software systems for multiple architectures, operating systems, and configurations. A set of inconsistent or missing compiler flags generates code that catastrophically impacts the system's behavior. In the authors' industry experience, defects caused by an undesired combination of compiler flags are common in nontrivial software projects. We are unaware of any build and CI/C… ▽ More Engineers build large software systems for multiple architectures, operating systems, and configurations. A set of inconsistent or missing compiler flags generates code that catastrophically impacts the system's behavior. In the authors' industry experience, defects caused by an undesired combination of compiler flags are common in nontrivial software projects. We are unaware of any build and CI/CD systems that track how the compiler produces a specific binary in a structured manner. We postulate that a queryable database of how the compiler compiled and linked the software system will help to detect defects earlier and reduce the debugging time. △ Less

Submitted 20 December, 2023; originally announced December 2023.

Comments: 3 pages. To be published in the 46th International Conference on Software Engineering (ICSE 2024), April 14 - April 20 2024, Lisbon, Portugal

What Do You Mean by Memory? When Engineers Are Lost in the Maze of Complexity

Authors: Gunnar Kudrjavets, Aditya Kumar, Jeff Thomas, Ayushi Rastogi

Abstract: An accepted practice to decrease applications' memory usage is to reduce the amount and frequency of memory allocations. Factors such as (a) the prevalence of out-of-memory (OOM) killers, (b) memory allocations in modern programming languages done implicitly, (c) overcommitting being a default strategy in the Linux kernel, and (d) the rise in complexity and terminology related to memory management… ▽ More An accepted practice to decrease applications' memory usage is to reduce the amount and frequency of memory allocations. Factors such as (a) the prevalence of out-of-memory (OOM) killers, (b) memory allocations in modern programming languages done implicitly, (c) overcommitting being a default strategy in the Linux kernel, and (d) the rise in complexity and terminology related to memory management makes the existing guidance inefficient. The industry needs detailed guidelines for optimizing memory usage targeting specific operating systems (OS) and programming language types. △ Less

Submitted 20 December, 2023; originally announced December 2023.

Comments: 3 pages. To be published in the 46th International Conference on Software Engineering (ICSE 2024), April 14 - April 20 2024, Lisbon, Portugal

Can we Agree? On the Rashōmon Effect and the Reliability of Post-Hoc Explainable AI

Authors: Clement Poiret, Antoine Grigis, Justin Thomas, Marion Noulhiane

Abstract: The Rashōmon effect poses challenges for deriving reliable knowledge from machine learning models. This study examined the influence of sample size on explanations from models in a Rashōmon set using SHAP. Experiments on 5 public datasets showed that explanations gradually converged as the sample size increased. Explanations from <128 samples exhibited high variability, limiting reliable knowledge… ▽ More The Rashōmon effect poses challenges for deriving reliable knowledge from machine learning models. This study examined the influence of sample size on explanations from models in a Rashōmon set using SHAP. Experiments on 5 public datasets showed that explanations gradually converged as the sample size increased. Explanations from <128 samples exhibited high variability, limiting reliable knowledge extraction. However, agreement between models improved with more data, allowing for consensus. Bagging ensembles often had higher agreement. The results provide guidance on sufficient data to trust explanations. Variability at low samples suggests that conclusions may be unreliable without validation. Further work is needed with more model types, data domains, and explanation methods. Testing convergence in neural networks and with model-specific explanation methods would be impactful. The approaches explored here point towards principled techniques for eliciting knowledge from ambiguous models. △ Less

Submitted 14 August, 2023; originally announced August 2023.

Comments: 13 pages, 6 figures and 6 tables

ACM Class: H.1.1; I.6.4; I.2.1

Joint speech and overlap detection: a benchmark over multiple audio setup and speech domains

Authors: Martin Lebourdais, Théo Mariotte, Marie Tahon, Anthony Larcher, Antoine Laurent, Silvio Montresor, Sylvain Meignier, Jean-Hugh Thomas

Abstract: Voice activity and overlapped speech detection (respectively VAD and OSD) are key pre-processing tasks for speaker diarization. The final segmentation performance highly relies on the robustness of these sub-tasks. Recent studies have shown VAD and OSD can be trained jointly using a multi-class classification model. However, these works are often restricted to a specific speech domain, lacking inf… ▽ More Voice activity and overlapped speech detection (respectively VAD and OSD) are key pre-processing tasks for speaker diarization. The final segmentation performance highly relies on the robustness of these sub-tasks. Recent studies have shown VAD and OSD can be trained jointly using a multi-class classification model. However, these works are often restricted to a specific speech domain, lacking information about the generalization capacities of the systems. This paper proposes a complete and new benchmark of different VAD and OSD models, on multiple audio setups (single/multi-channel) and speech domains (e.g. media, meeting...). Our 2/3-class systems, which combine a Temporal Convolutional Network with speech representations adapted to the setup, outperform state-of-the-art results. We show that the joint training of these two tasks offers similar performances in terms of F1-score to two dedicated VAD and OSD systems while reducing the training cost. This unique architecture can also be used for single and multichannel speech processing. △ Less

Submitted 24 July, 2023; originally announced July 2023.

Multi-Objective Optimization of Performance and Interpretability of Tabular Supervised Machine Learning Models

Authors: Lennart Schneider, Bernd Bischl, Janek Thomas

Abstract: We present a model-agnostic framework for jointly optimizing the predictive performance and interpretability of supervised machine learning models for tabular data. Interpretability is quantified via three measures: feature sparsity, interaction sparsity of features, and sparsity of non-monotone feature effects. By treating hyperparameter optimization of a machine learning algorithm as a multi-obj… ▽ More We present a model-agnostic framework for jointly optimizing the predictive performance and interpretability of supervised machine learning models for tabular data. Interpretability is quantified via three measures: feature sparsity, interaction sparsity of features, and sparsity of non-monotone feature effects. By treating hyperparameter optimization of a machine learning algorithm as a multi-objective optimization problem, our framework allows for generating diverse models that trade off high performance and ease of interpretability in a single optimization run. Efficient optimization is achieved via augmentation of the search space of the learning algorithm by incorporating feature selection, interaction and monotonicity constraints into the hyperparameter search space. We demonstrate that the optimization problem effectively translates to finding the Pareto optimal set of groups of selected features that are allowed to interact in a model, along with finding their optimal monotonicity constraints and optimal hyperparameters of the learning algorithm itself. We then introduce a novel evolutionary algorithm that can operate efficiently on this augmented search space. In benchmark experiments, we show that our framework is capable of finding diverse models that are highly competitive or outperform state-of-the-art XGBoost or Explainable Boosting Machine models, both with respect to performance and interpretability. △ Less

Submitted 16 July, 2023; originally announced July 2023.

Comments: Extended version of the paper accepted at GECCO 2023. 16 pages, 7 tables, 7 figures

Multi-microphone Automatic Speech Segmentation in Meetings Based on Circular Harmonics Features

Authors: Théo Mariotte, Anthony Larcher, Silvio Montrésor, Jean-Hugh Thomas

Abstract: Speaker diarization is the task of answering Who spoke and when? in an audio stream. Pipeline systems rely on speech segmentation to extract speakers' segments and achieve robust speaker diarization. This paper proposes a common framework to solve three segmentation tasks in the distant speech scenario: Voice Activity Detection (VAD), Overlapped Speech Detection (OSD), and Speaker Change Detection… ▽ More Speaker diarization is the task of answering Who spoke and when? in an audio stream. Pipeline systems rely on speech segmentation to extract speakers' segments and achieve robust speaker diarization. This paper proposes a common framework to solve three segmentation tasks in the distant speech scenario: Voice Activity Detection (VAD), Overlapped Speech Detection (OSD), and Speaker Change Detection (SCD). In the literature, a few studies investigate the multi-microphone distant speech scenario. In this work, we propose a new set of spatial features based on direction-of-arrival estimations in the circular harmonic domain (CH-DOA). These spatial features are extracted from multi-microphone audio data and combined with standard acoustic features. Experiments on the AMI meeting corpus show that CH-DOA can improve the segmentation while being robust in the case of deactivated microphones. △ Less

Submitted 7 June, 2023; originally announced June 2023.

Comments: Interspeech 2023, international Speech Communication Association (ISCA), Aug 2023, Dublin, Ireland

Unsupervised Spiking Neural Network Model of Prefrontal Cortex to study Task Switching with Synaptic deficiency

Authors: Ashwin Viswanathan Kannan, Goutam Mylavarapu, Johnson P Thomas

Abstract: In this study, we build a computational model of Prefrontal Cortex (PFC) using Spiking Neural Networks (SNN) to understand how neurons adapt and respond to tasks switched under short and longer duration of stimulus changes. We also explore behavioral deficits arising out of the PFC lesions by simulating lesioned states in our Spiking architecture model. Although there are some computational models… ▽ More In this study, we build a computational model of Prefrontal Cortex (PFC) using Spiking Neural Networks (SNN) to understand how neurons adapt and respond to tasks switched under short and longer duration of stimulus changes. We also explore behavioral deficits arising out of the PFC lesions by simulating lesioned states in our Spiking architecture model. Although there are some computational models of the PFC, SNN's have not been used to model them. In this study, we use SNN's having parameters close to biologically plausible values and train the model using unsupervised Spike Timing Dependent Plasticity (STDP) learning rule. Our model is based on connectionist architectures and exhibits neural phenomena like sustained activity which helps in generating short-term or working memory. We use these features to simulate lesions by deactivating synaptic pathways and record the weight adjustments of learned patterns and capture the accuracy of learning tasks in such conditions. All our experiments are trained and recorded using a real-world Fashion MNIST (FMNIST) dataset and through this work, we bridge the gap between bio-realistic models and those that perform well in pattern recognition tasks △ Less

Submitted 23 May, 2023; originally announced May 2023.

MO-DEHB: Evolutionary-based Hyperband for Multi-Objective Optimization

Authors: Noor Awad, Ayushi Sharma, Philipp Muller, Janek Thomas, Frank Hutter

Abstract: Hyperparameter optimization (HPO) is a powerful technique for automating the tuning of machine learning (ML) models. However, in many real-world applications, accuracy is only one of multiple performance criteria that must be considered. Optimizing these objectives simultaneously on a complex and diverse search space remains a challenging task. In this paper, we propose MO-DEHB, an effective and f… ▽ More Hyperparameter optimization (HPO) is a powerful technique for automating the tuning of machine learning (ML) models. However, in many real-world applications, accuracy is only one of multiple performance criteria that must be considered. Optimizing these objectives simultaneously on a complex and diverse search space remains a challenging task. In this paper, we propose MO-DEHB, an effective and flexible multi-objective (MO) optimizer that extends the recent evolutionary Hyperband method DEHB. We validate the performance of MO-DEHB using a comprehensive suite of 15 benchmarks consisting of diverse and challenging MO problems, including HPO, neural architecture search (NAS), and joint NAS and HPO, with objectives including accuracy, latency and algorithmic fairness. A comparative study against state-of-the-art MO optimizers demonstrates that MO-DEHB clearly achieves the best performance across our 15 benchmarks. △ Less

Submitted 11 May, 2023; v1 submitted 8 May, 2023; originally announced May 2023.

Identifying Symptoms of Delirium from Clinical Narratives Using Natural Language Processing

Authors: Aokun Chen, Daniel Paredes, Zehao Yu, Xiwei Lou, Roberta Brunson, Jamie N. Thomas, Kimberly A. Martinez, Robert J. Lucero, Tanja Magoc, Laurence M. Solberg, Urszula A. Snigurska, Sarah E. Ser, Mattia Prosperi, Jiang Bian, Ragnhildur I. Bjarnadottir, Yonghui Wu

Abstract: Delirium is an acute decline or fluctuation in attention, awareness, or other cognitive function that can lead to serious adverse outcomes. Despite the severe outcomes, delirium is frequently unrecognized and uncoded in patients' electronic health records (EHRs) due to its transient and diverse nature. Natural language processing (NLP), a key technology that extracts medical concepts from clinical… ▽ More Delirium is an acute decline or fluctuation in attention, awareness, or other cognitive function that can lead to serious adverse outcomes. Despite the severe outcomes, delirium is frequently unrecognized and uncoded in patients' electronic health records (EHRs) due to its transient and diverse nature. Natural language processing (NLP), a key technology that extracts medical concepts from clinical narratives, has shown great potential in studies of delirium outcomes and symptoms. To assist in the diagnosis and phenotyping of delirium, we formed an expert panel to categorize diverse delirium symptoms, composed annotation guidelines, created a delirium corpus with diverse delirium symptoms, and developed NLP methods to extract delirium symptoms from clinical notes. We compared 5 state-of-the-art transformer models including 2 models (BERT and RoBERTa) from the general domain and 3 models (BERT_MIMIC, RoBERTa_MIMIC, and GatorTron) from the clinical domain. GatorTron achieved the best strict and lenient F1 scores of 0.8055 and 0.8759, respectively. We conducted an error analysis to identify challenges in annotating delirium symptoms and developing NLP systems. To the best of our knowledge, this is the first large language model-based delirium symptom extraction system. Our study lays the foundation for the future development of computable phenotypes and diagnosis methods for delirium. △ Less

Submitted 31 March, 2023; originally announced April 2023.

Investigating Strategies for Clause Recommendation

Authors: Sagar Joshi, Sumanth Balaji, Jerrin Thomas, Aparna Garimella, Vasudeva Varma

Abstract: Clause recommendation is the problem of recommending a clause to a legal contract, given the context of the contract in question and the clause type to which the clause should belong. With not much prior work being done toward the generation of legal contracts, this problem was proposed as a first step toward the bigger problem of contract generation. As an open-ended text generation problem, the… ▽ More Clause recommendation is the problem of recommending a clause to a legal contract, given the context of the contract in question and the clause type to which the clause should belong. With not much prior work being done toward the generation of legal contracts, this problem was proposed as a first step toward the bigger problem of contract generation. As an open-ended text generation problem, the distinguishing characteristics of this problem lie in the nature of legal language as a sublanguage and the considerable similarity of textual content within the clauses of a specific type. This similarity aspect in legal clauses drives us to investigate the importance of similar contracts' representation for recommending clauses. In our work, we experiment with generating clauses for 15 commonly occurring clause types in contracts expanding upon the previous work on this problem and analyzing clause recommendations in varying settings using information derived from similar contracts. △ Less

Submitted 21 January, 2023; originally announced January 2023.

Comments: Published in Legal Knowledge and Information Systems (JURIX) 2022. (10 pages, 4 figures)

ACM Class: I.2.7

Journal ref: Volume 362: Legal Knowledge and Information Systems (2022), Frontiers in Artificial Intelligence and Applications

Explicitly Solvable Continuous-time Inference for Partially Observed Markov Processes

Authors: Daniel Chen, Alexander G. Strang, Andrew W. Eckford, Peter J. Thomas

Abstract: Many natural and engineered systems can be modeled as discrete state Markov processes. Often, only a subset of states are directly observable. Inferring the conditional probability that a system occupies a particular hidden state, given the partial observation, is a problem with broad application. In this paper, we introduce a continuous-time formulation of the sum-product algorithm, which is a we… ▽ More Many natural and engineered systems can be modeled as discrete state Markov processes. Often, only a subset of states are directly observable. Inferring the conditional probability that a system occupies a particular hidden state, given the partial observation, is a problem with broad application. In this paper, we introduce a continuous-time formulation of the sum-product algorithm, which is a well-known discrete-time method for finding the hidden states' conditional probabilities, given a set of finite, discrete-time observations. From our new formulation, we can explicitly solve for the conditional probability of occupying any state, given the transition rates and observations within a finite time window. We apply our algorithm to a realistic model of the cystic fibrosis transmembrane conductance regulator (CFTR) protein for exact inference of the conditional occupancy probability, given a finite time series of partial observations. △ Less

Submitted 2 January, 2023; originally announced January 2023.

Comments: Accepted for publication in IEEE Transactions on Signal Processing

Who Ate My Memory? Towards Attribution in Memory Management

Authors: Gunnar Kudrjavets, Ayushi Rastogi, Jeff Thomas, Nachiappan Nagappan

Abstract: To understand applications' memory usage details, engineers use instrumented builds and profiling tools. Both approaches are impractical for use in production environments or deployed mobile applications. As a result, developers can gather only high-level memory-related statistics for deployed software. In our experience, the lack of granular field data makes fixing performance and reliability-rel… ▽ More To understand applications' memory usage details, engineers use instrumented builds and profiling tools. Both approaches are impractical for use in production environments or deployed mobile applications. As a result, developers can gather only high-level memory-related statistics for deployed software. In our experience, the lack of granular field data makes fixing performance and reliability-related defects complex and time-consuming. The software industry needs lightweight solutions to collect detailed data about applications' memory usage to increase developer productivity. Current research into memory attribution-related data structures, techniques, and tools is in the early stages and enables several new research avenues. △ Less

Submitted 22 December, 2022; originally announced December 2022.

Comments: 3 pages. To be published in the 45th International Conference on Software Engineering (ICSE 2023), May 14 - May 20 2023, Melbourne, Australia

Scalable Multi-Agent Reinforcement Learning for Warehouse Logistics with Robotic and Human Co-Workers

Authors: Aleksandar Krnjaic, Raul D. Steleac, Jonathan D. Thomas, Georgios Papoudakis, Lukas Schäfer, Andrew Wing Keung To, Kuan-Ho Lao, Murat Cubuktepe, Matthew Haley, Peter Börsting, Stefano V. Albrecht

Abstract: We consider a warehouse in which dozens of mobile robots and human pickers work together to collect and deliver items within the warehouse. The fundamental problem we tackle, called the order-picking problem, is how these worker agents must coordinate their movement and actions in the warehouse to maximise performance in this task. Established industry methods using heuristic approaches require la… ▽ More We consider a warehouse in which dozens of mobile robots and human pickers work together to collect and deliver items within the warehouse. The fundamental problem we tackle, called the order-picking problem, is how these worker agents must coordinate their movement and actions in the warehouse to maximise performance in this task. Established industry methods using heuristic approaches require large engineering efforts to optimise for innately variable warehouse configurations. In contrast, multi-agent reinforcement learning (MARL) can be flexibly applied to diverse warehouse configurations (e.g. size, layout, number/types of workers, item replenishment frequency), and different types of order-picking paradigms (e.g. Goods-to-Person and Person-to-Goods), as the agents can learn how to cooperate optimally through experience. We develop hierarchical MARL algorithms in which a manager agent assigns goals to worker agents, and the policies of the manager and workers are co-trained toward maximising a global objective (e.g. pick rate). Our hierarchical algorithms achieve significant gains in sample efficiency over baseline MARL algorithms and overall pick rates over multiple established industry heuristics in a diverse set of warehouse configurations and different order-picking paradigms. △ Less

Submitted 30 August, 2024; v1 submitted 22 December, 2022; originally announced December 2022.

Comments: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2024

Weisfeiler-Lehman goes Dynamic: An Analysis of the Expressive Power of Graph Neural Networks for Attributed and Dynamic Graphs

Authors: Silvia Beddar-Wiesing, Giuseppe Alessio D'Inverno, Caterina Graziani, Veronica Lachi, Alice Moallemy-Oureh, Franco Scarselli, Josephine Maria Thomas

Abstract: Graph Neural Networks (GNNs) are a large class of relational models for graph processing. Recent theoretical studies on the expressive power of GNNs have focused on two issues. On the one hand, it has been proven that GNNs are as powerful as the Weisfeiler-Lehman test (1-WL) in their ability to distinguish graphs. Moreover, it has been shown that the equivalence enforced by 1-WL equals unfolding e… ▽ More Graph Neural Networks (GNNs) are a large class of relational models for graph processing. Recent theoretical studies on the expressive power of GNNs have focused on two issues. On the one hand, it has been proven that GNNs are as powerful as the Weisfeiler-Lehman test (1-WL) in their ability to distinguish graphs. Moreover, it has been shown that the equivalence enforced by 1-WL equals unfolding equivalence. On the other hand, GNNs turned out to be universal approximators on graphs modulo the constraints enforced by 1-WL/unfolding equivalence. However, these results only apply to Static Attributed Undirected Homogeneous Graphs (SAUHG) with node attributes. In contrast, real-life applications often involve a much larger variety of graph types. In this paper, we conduct a theoretical analysis of the expressive power of GNNs for two other graph domains that are particularly interesting in practical applications, namely dynamic graphs and SAUGHs with edge attributes. Dynamic graphs are widely used in modern applications; hence, the study of the expressive capability of GNNs in this domain is essential for practical reasons and, in addition, it requires a new analyzing approach due to the difference in the architecture of dynamic GNNs compared to static ones. On the other hand, the examination of SAUHGs is of particular relevance since they act as a standard form for all graph types: it has been shown that all graph types can be transformed without loss of information to SAUHGs with both attributes on nodes and edges. This paper considers generic GNN models and appropriate 1-WL tests for those domains. Then, the known results on the expressive power of GNNs are extended to the mentioned domains: it is proven that GNNs have the same capability as the 1-WL test, the 1-WL equivalence equals unfolding equivalence and that GNNs are universal approximators modulo 1-WL/unfolding equivalence. △ Less

Submitted 3 May, 2024; v1 submitted 8 October, 2022; originally announced October 2022.

Federated Meta-Learning for Traffic Steering in O-RAN

Authors: Hakan Erdol, Xiaoyang Wang, Peizheng Li, Jonathan D. Thomas, Robert Piechocki, George Oikonomou, Rui Inacio, Abdelrahim Ahmad, Keith Briggs, Shipra Kapoor

Abstract: The vision of 5G lies in providing high data rates, low latency (for the aim of near-real-time applications), significantly increased base station capacity, and near-perfect quality of service (QoS) for users, compared to LTE networks. In order to provide such services, 5G systems will support various combinations of access technologies such as LTE, NR, NR-U and Wi-Fi. Each radio access technology… ▽ More The vision of 5G lies in providing high data rates, low latency (for the aim of near-real-time applications), significantly increased base station capacity, and near-perfect quality of service (QoS) for users, compared to LTE networks. In order to provide such services, 5G systems will support various combinations of access technologies such as LTE, NR, NR-U and Wi-Fi. Each radio access technology (RAT) provides different types of access, and these should be allocated and managed optimally among the users. Besides resource management, 5G systems will also support a dual connectivity service. The orchestration of the network therefore becomes a more difficult problem for system managers with respect to legacy access technologies. In this paper, we propose an algorithm for RAT allocation based on federated meta-learning (FML), which enables RAN intelligent controllers (RICs) to adapt more quickly to dynamically changing environments. We have designed a simulation environment which contains LTE and 5G NR service technologies. In the simulation, our objective is to fulfil UE demands within the deadline of transmission to provide higher QoS values. We compared our proposed algorithm with a single RL agent, the Reptile algorithm and a rule-based heuristic method. Simulation results show that the proposed FML method achieves higher caching rates at first deployment round 21% and 12% respectively. Moreover, proposed approach adapts to new tasks and environments most quickly amongst the compared methods. △ Less

Submitted 13 September, 2022; originally announced September 2022.

Comments: 7 pages, 3 figures, 2 algorithms, and 3 tables

When malloc() Never Returns NULL -- Reliability as an Illusion

Authors: Gunnar Kudrjavets, Jeff Thomas, Aditya Kumar, Nachiappan Nagappan, Ayushi Rastogi

Abstract: For decades, the guidance given to software engineers has been to check the memory allocation results. This validation step is necessary to avoid crashes. However, in user mode, in modern operating systems (OS), such as Android, FreeBSD, iOS, and macOS, the caller does not have an opportunity to handle the memory allocation failures. This behavioral trait results from the actions of a system compo… ▽ More For decades, the guidance given to software engineers has been to check the memory allocation results. This validation step is necessary to avoid crashes. However, in user mode, in modern operating systems (OS), such as Android, FreeBSD, iOS, and macOS, the caller does not have an opportunity to handle the memory allocation failures. This behavioral trait results from the actions of a system component called an out-of-memory (OOM) killer. We identify that the only mainstream OS that, by default, lets applications detect memory allocation failures is Microsoft Windows. The false expectation that an application can handle OOM errors can negatively impact its design. The presence of error-handling code creates an illusion of reliability and is wasteful in terms of lines of code and code size. We describe the current behavior of a sample of popular OSs during low-memory conditions and provide recommendations for engineering practices going forward. △ Less

Submitted 17 August, 2022; originally announced August 2022.

Comments: 6 pages. To be published in the 33rd IEEE International Symposium on Software Reliability Engineering (ISSRE 2022), Oct 31 - Nov 3 2022, Charlotte, North Carolina, USA

Tackling Neural Architecture Search With Quality Diversity Optimization

Authors: Lennart Schneider, Florian Pfisterer, Paul Kent, Juergen Branke, Bernd Bischl, Janek Thomas

Abstract: Neural architecture search (NAS) has been studied extensively and has grown to become a research field with substantial impact. While classical single-objective NAS searches for the architecture with the best performance, multi-objective NAS considers multiple objectives that should be optimized simultaneously, e.g., minimizing resource usage along the validation error. Although considerable progr… ▽ More Neural architecture search (NAS) has been studied extensively and has grown to become a research field with substantial impact. While classical single-objective NAS searches for the architecture with the best performance, multi-objective NAS considers multiple objectives that should be optimized simultaneously, e.g., minimizing resource usage along the validation error. Although considerable progress has been made in the field of multi-objective NAS, we argue that there is some discrepancy between the actual optimization problem of practical interest and the optimization problem that multi-objective NAS tries to solve. We resolve this discrepancy by formulating the multi-objective NAS problem as a quality diversity optimization (QDO) problem and introduce three quality diversity NAS optimizers (two of them belonging to the group of multifidelity optimizers), which search for high-performing yet diverse architectures that are optimal for application-specific niches, e.g., hardware constraints. By comparing these optimizers to their multi-objective counterparts, we demonstrate that quality diversity NAS in general outperforms multi-objective NAS with respect to quality of solutions and efficiency. We further show how applications and future NAS research can thrive on QDO. △ Less

Submitted 30 July, 2022; originally announced August 2022.

Comments: Accepted at the First Conference on Automated Machine Learning (Main Track). 30 pages, 8 tables, 13 figures

Learning Feature Decomposition for Domain Adaptive Monocular Depth Estimation

Authors: Shao-Yuan Lo, Wei Wang, Jim Thomas, Jingjing Zheng, Vishal M. Patel, Cheng-Hao Kuo

Abstract: Monocular depth estimation (MDE) has attracted intense study due to its low cost and critical functions for robotic tasks such as localization, mapping and obstacle detection. Supervised approaches have led to great success with the advance of deep learning, but they rely on large quantities of ground-truth depth annotations that are expensive to acquire. Unsupervised domain adaptation (UDA) trans… ▽ More Monocular depth estimation (MDE) has attracted intense study due to its low cost and critical functions for robotic tasks such as localization, mapping and obstacle detection. Supervised approaches have led to great success with the advance of deep learning, but they rely on large quantities of ground-truth depth annotations that are expensive to acquire. Unsupervised domain adaptation (UDA) transfers knowledge from labeled source data to unlabeled target data, so as to relax the constraint of supervised learning. However, existing UDA approaches may not completely align the domain gap across different datasets because of the domain shift problem. We believe better domain alignment can be achieved via well-designed feature decomposition. In this paper, we propose a novel UDA method for MDE, referred to as Learning Feature Decomposition for Adaptation (LFDA), which learns to decompose the feature space into content and style components. LFDA only attempts to align the content component since it has a smaller domain gap. Meanwhile, it excludes the style component which is specific to the source domain from training the primary task. Furthermore, LFDA uses separate feature distribution estimations to further bridge the domain gap. Extensive experiments on three domain adaptative MDE scenarios show that the proposed method achieves superior accuracy and lower computational cost compared to the state-of-the-art approaches. △ Less

Submitted 30 July, 2022; originally announced August 2022.

Comments: Accepted at IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2022

AMLB: an AutoML Benchmark

Authors: Pieter Gijsbers, Marcos L. P. Bueno, Stefan Coors, Erin LeDell, Sébastien Poirier, Janek Thomas, Bernd Bischl, Joaquin Vanschoren

Abstract: Comparing different AutoML frameworks is notoriously challenging and often done incorrectly. We introduce an open and extensible benchmark that follows best practices and avoids common mistakes when comparing AutoML frameworks. We conduct a thorough comparison of 9 well-known AutoML frameworks across 71 classification and 33 regression tasks. The differences between the AutoML frameworks are explo… ▽ More Comparing different AutoML frameworks is notoriously challenging and often done incorrectly. We introduce an open and extensible benchmark that follows best practices and avoids common mistakes when comparing AutoML frameworks. We conduct a thorough comparison of 9 well-known AutoML frameworks across 71 classification and 33 regression tasks. The differences between the AutoML frameworks are explored with a multi-faceted analysis, evaluating model accuracy, its trade-offs with inference time, and framework failures. We also use Bradley-Terry trees to discover subsets of tasks where the relative AutoML framework rankings differ. The benchmark comes with an open-source tool that integrates with many AutoML frameworks and automates the empirical evaluation process end-to-end: from framework installation and resource allocation to in-depth evaluation. The benchmark uses public data sets, can be easily extended with other AutoML frameworks and tasks, and has a website with up-to-date results. △ Less

Submitted 16 November, 2023; v1 submitted 25 July, 2022; originally announced July 2022.

Comments: UNDER REVIEW: Revised submission to JMLR, with updated results from June 2023

There Ain't No Such Thing as a Free Custom Memory Allocator

Authors: Gunnar Kudrjavets, Jeff Thomas, Aditya Kumar, Nachiappan Nagappan, Ayushi Rastogi

Abstract: Using custom memory allocators is an efficient performance optimization technique. However, dependency on a custom allocator can introduce several maintenance-related issues. We present lessons learned from the industry and provide critical guidance for using custom memory allocators and enumerate various challenges associated with integrating them. These recommendations are based on years of expe… ▽ More Using custom memory allocators is an efficient performance optimization technique. However, dependency on a custom allocator can introduce several maintenance-related issues. We present lessons learned from the industry and provide critical guidance for using custom memory allocators and enumerate various challenges associated with integrating them. These recommendations are based on years of experience incorporating custom allocators into different industrial software projects. △ Less

Submitted 23 June, 2022; originally announced June 2022.

Comments: 4 pages. To be published in 38th IEEE International Conference on Software Maintenance and Evolution (ICSME 2022), Oct 3-7, 2022, Limassol, Cyprus

Multi-Objective Hyperparameter Optimization in Machine Learning -- An Overview

Authors: Florian Karl, Tobias Pielok, Julia Moosbauer, Florian Pfisterer, Stefan Coors, Martin Binder, Lennart Schneider, Janek Thomas, Jakob Richter, Michel Lang, Eduardo C. Garrido-Merchán, Juergen Branke, Bernd Bischl

Abstract: Hyperparameter optimization constitutes a large part of typical modern machine learning workflows. This arises from the fact that machine learning methods and corresponding preprocessing steps often only yield optimal performance when hyperparameters are properly tuned. But in many applications, we are not only interested in optimizing ML pipelines solely for predictive accuracy; additional metric… ▽ More Hyperparameter optimization constitutes a large part of typical modern machine learning workflows. This arises from the fact that machine learning methods and corresponding preprocessing steps often only yield optimal performance when hyperparameters are properly tuned. But in many applications, we are not only interested in optimizing ML pipelines solely for predictive accuracy; additional metrics or constraints must be considered when determining an optimal configuration, resulting in a multi-objective optimization problem. This is often neglected in practice, due to a lack of knowledge and readily available software implementations for multi-objective hyperparameter optimization. In this work, we introduce the reader to the basics of multi-objective hyperparameter optimization and motivate its usefulness in applied ML. Furthermore, we provide an extensive survey of existing optimization strategies, both from the domain of evolutionary algorithms and Bayesian optimization. We illustrate the utility of MOO in several specific ML applications, considering objectives such as operating conditions, prediction time, sparseness, fairness, interpretability and robustness. △ Less

Submitted 6 June, 2024; v1 submitted 15 June, 2022; originally announced June 2022.

Comments: Published at ACM TELO

Journal ref: ACM Transactions on Evolutionary Learning and Optimization 3.4 (2023): 1-50

Is Kernel Code Different From Non-Kernel Code? A Case Study of BSD Family Operating Systems

Authors: Gunnar Kudrjavets, Jeff Thomas, Nachiappan Nagappan, Ayushi Rastogi

Abstract: Code churn and code velocity describe the evolution of a code base. Current research quantifies and studies code churn and velocity at a high level of abstraction, often at the overall project level or even at the level of an entire company. We argue that such an approach ignores noticeable differences among the subsystems of large projects. We conducted an exploratory study on four BSD family ope… ▽ More Code churn and code velocity describe the evolution of a code base. Current research quantifies and studies code churn and velocity at a high level of abstraction, often at the overall project level or even at the level of an entire company. We argue that such an approach ignores noticeable differences among the subsystems of large projects. We conducted an exploratory study on four BSD family operating systems: DragonFlyBSD, FreeBSD, NetBSD, and OpenBSD. We mine 797,879 commits to characterize code churn in terms of the annual growth rate, commit types, change type ratio, and size taxonomy of commits for different subsystems (kernel, non-kernel, and mixed). We also investigate differences among various code review periods, i.e., time-to-first-response, time-to-accept, and time-to-merge, as indicators of code velocity. Our study provides empirical evidence that quantifiable evolutionary code characteristics at a global system scope fail to take into account significant individual differences that exist at a subsystem level. We found that while there exist similarities in the code base growth rate and distribution of commit types (neutral, additive, and subtractive) across BSD subsystems, (a) most commits contain kernel or non-kernel code exclusively, (b) kernel commits are larger than non-kernel commits, and (c) code reviews for kernel code take longer than non-kernel code. △ Less

Submitted 11 June, 2022; originally announced June 2022.

Comments: 13 pages. To be published in 38th IEEE International Conference on Software Maintenance and Evolution (ICSME 2022), Oct 3-7, 2022, Limassol, Cyprus

Sim2real for Reinforcement Learning Driven Next Generation Networks

Authors: Peizheng Li, Jonathan Thomas, Xiaoyang Wang, Hakan Erdol, Abdelrahim Ahmad, Rui Inacio, Shipra Kapoor, Arjun Parekh, Angela Doufexi, Arman Shojaeifard, Robert Piechocki

Abstract: The next generation of networks will actively embrace artificial intelligence (AI) and machine learning (ML) technologies for automation networks and optimal network operation strategies. The emerging network structure represented by Open RAN (O-RAN) conforms to this trend, and the radio intelligent controller (RIC) at the centre of its specification serves as an ML applications host. Various ML m… ▽ More The next generation of networks will actively embrace artificial intelligence (AI) and machine learning (ML) technologies for automation networks and optimal network operation strategies. The emerging network structure represented by Open RAN (O-RAN) conforms to this trend, and the radio intelligent controller (RIC) at the centre of its specification serves as an ML applications host. Various ML models, especially Reinforcement Learning (RL) models, are regarded as the key to solving RAN-related multi-objective optimization problems. However, it should be recognized that most of the current RL successes are confined to abstract and simplified simulation environments, which may not directly translate to high performance in complex real environments. One of the main reasons is the modelling gap between the simulation and the real environment, which could make the RL agent trained by simulation ill-equipped for the real environment. This issue is termed as the sim2real gap. This article brings to the fore the sim2real challenge within the context of O-RAN. Specifically, it emphasizes the characteristics, and benefits that the digital twins (DT) could have as a place for model development and verification. Several use cases are presented to exemplify and demonstrate failure modes of the simulations trained RL model in real environments. The effectiveness of DT in assisting the development of RL algorithms is discussed. Then the current state of the art learning-based methods commonly used to overcome the sim2real challenge are presented. Finally, the development and deployment concerns for the RL applications realisation in O-RAN are discussed from the view of the potential issues like data interaction, environment bottlenecks, and algorithm design. △ Less

Submitted 8 June, 2022; originally announced June 2022.

Comments: 7 pages, 4 figures

Marked Neural Spatio-Temporal Point Process Involving a Dynamic Graph Neural Network

Authors: Alice Moallemy-Oureh, Silvia Beddar-Wiesing, Yannick Nagel, Rüdiger Nather, Josephine M. Thomas

Abstract: Temporal Point Processes (TPPs) have recently become increasingly interesting for learning dynamics in graph data. A reason for this is that learning on dynamic graph data is becoming more relevant, since data from many scientific fields, ranging from mathematics, biology, social sciences, and physics to computer science, is naturally related and inherently dynamic. In addition, TPPs provide a mea… ▽ More Temporal Point Processes (TPPs) have recently become increasingly interesting for learning dynamics in graph data. A reason for this is that learning on dynamic graph data is becoming more relevant, since data from many scientific fields, ranging from mathematics, biology, social sciences, and physics to computer science, is naturally related and inherently dynamic. In addition, TPPs provide a meaningful characterization of event streams and a prediction mechanism for future events. Therefore, (semi-)parameterized Neural TPPs have been introduced whose characterization can be (partially) learned and, thus, enable the representation of more complex phenomena. However, the research on modeling dynamic graphs with TPPs is relatively young, and only a few models for node attribute changes or evolving edges have been proposed yet. To allow for learning on fully dynamic graph streams, i.e., graphs that can change in their structure (addition/deletion of nodes/edge) and in their node/edge attributes, we propose a Marked Neural Spatio-Temporal Point Process (MNSTPP). It leverages a Dynamic Graph Neural Network to learn a Marked TPP that handles attributes and spatial data to model and predict any event in a graph stream. △ Less

Submitted 28 August, 2024; v1 submitted 7 June, 2022; originally announced June 2022.

Achieving Goals using Reward Shaping and Curriculum Learning

Authors: Mihai Anca, Jonathan D. Thomas, Dabal Pedamonti, Matthew Studley, Mark Hansen

Abstract: Real-time control for robotics is a popular research area in the reinforcement learning community. Through the use of techniques such as reward shaping, researchers have managed to train online agents across a multitude of domains. Despite these advances, solving goal-oriented tasks still requires complex architectural changes or hard constraints to be placed on the problem. In this article, we so… ▽ More Real-time control for robotics is a popular research area in the reinforcement learning community. Through the use of techniques such as reward shaping, researchers have managed to train online agents across a multitude of domains. Despite these advances, solving goal-oriented tasks still requires complex architectural changes or hard constraints to be placed on the problem. In this article, we solve the problem of stacking multiple cubes by combining curriculum learning, reward shaping, and a high number of efficiently parallelized environments. We introduce two curriculum learning settings that allow us to separate the complex task into sequential sub-goals, hence enabling the learning of a problem that may otherwise be too difficult. We focus on discussing the challenges encountered while implementing them in a goal-conditioned environment. Finally, we extend the best configuration identified on a higher complexity environment with differently shaped objects. △ Less

Submitted 20 April, 2023; v1 submitted 6 June, 2022; originally announced June 2022.

Comments: To be published at Future Technologies Conference (FTC) 2023

The Evolving Landscape of Software Performance Engineering

Authors: Gunnar Kudrjavets, Jeff Thomas, Nachiappan Nagappan

Abstract: Satisfactory software performance is essential for the adoption and the success of a product. In organizations that follow traditional software development models (e.g., waterfall), Software Performance Engineering (SPE) involves time-consuming experimental modeling and performance testing outside the actual production environment. Such existing SPE methods, however, are not optimized for environm… ▽ More Satisfactory software performance is essential for the adoption and the success of a product. In organizations that follow traditional software development models (e.g., waterfall), Software Performance Engineering (SPE) involves time-consuming experimental modeling and performance testing outside the actual production environment. Such existing SPE methods, however, are not optimized for environments utilizing Continuous Integration (CI) and Continuous Delivery (CD) that result in high frequency and high volume of code changes. We present a summary of lessons learned and propose improvements to the SPE process in the context of CI/CD. Our findings are based on SPE work on products A and B conducted over 5 years at an online services company X. We find that (a) SPE has mainly become a post hoc activity based on data from the production environment, (b) successful application of SPE techniques require frequent re-evaluation of priorities, and (c) engineers working on SPE require a broader skill set than one traditionally possessed by engineers working on performance. △ Less

Submitted 5 May, 2022; originally announced May 2022.

Comments: 2 pages. To be published in The International Conference on Evaluation and Assessment in Software Engineering 2022 (EASE 2022), June 13-15, 2022, Gothenburg, Sweden

A Collection of Quality Diversity Optimization Problems Derived from Hyperparameter Optimization of Machine Learning Models

Authors: Lennart Schneider, Florian Pfisterer, Janek Thomas, Bernd Bischl

Abstract: The goal of Quality Diversity Optimization is to generate a collection of diverse yet high-performing solutions to a given problem at hand. Typical benchmark problems are, for example, finding a repertoire of robot arm configurations or a collection of game playing strategies. In this paper, we propose a set of Quality Diversity Optimization problems that tackle hyperparameter optimization of mach… ▽ More The goal of Quality Diversity Optimization is to generate a collection of diverse yet high-performing solutions to a given problem at hand. Typical benchmark problems are, for example, finding a repertoire of robot arm configurations or a collection of game playing strategies. In this paper, we propose a set of Quality Diversity Optimization problems that tackle hyperparameter optimization of machine learning models - a so far underexplored application of Quality Diversity Optimization. Our benchmark problems involve novel feature functions, such as interpretability or resource usage of models. To allow for fast and efficient benchmarking, we build upon YAHPO Gym, a recently proposed open source benchmarking suite for hyperparameter optimization that makes use of high performing surrogate models and returns these surrogate model predictions instead of evaluating the true expensive black box function. We present results of an initial experimental study comparing different Quality Diversity optimizers on our benchmark problems. Furthermore, we discuss future directions and challenges of Quality Diversity Optimization in the context of hyperparameter optimization. △ Less

Submitted 30 July, 2022; v1 submitted 28 April, 2022; originally announced April 2022.

Comments: Accepted at the GECCO'22 Workshop on Quality Diversity Algorithm Benchmarks. 7 pages, 6 tables, 7 figures

Graph Neural Networks Designed for Different Graph Types: A Survey

Authors: Josephine M. Thomas, Alice Moallemy-Oureh, Silvia Beddar-Wiesing, Clara Holzhüter

Abstract: Graphs are ubiquitous in nature and can therefore serve as models for many practical but also theoretical problems. For this purpose, they can be defined as many different types which suitably reflect the individual contexts of the represented problem. To address cutting-edge problems based on graph data, the research field of Graph Neural Networks (GNNs) has emerged. Despite the field's youth and… ▽ More Graphs are ubiquitous in nature and can therefore serve as models for many practical but also theoretical problems. For this purpose, they can be defined as many different types which suitably reflect the individual contexts of the represented problem. To address cutting-edge problems based on graph data, the research field of Graph Neural Networks (GNNs) has emerged. Despite the field's youth and the speed at which new models are developed, many recent surveys have been published to keep track of them. Nevertheless, it has not yet been gathered which GNN can process what kind of graph types. In this survey, we give a detailed overview of already existing GNNs and, unlike previous surveys, categorize them according to their ability to handle different graph types and properties. We consider GNNs operating on static and dynamic graphs of different structural constitutions, with or without node or edge attributes. Moreover, we distinguish between GNN models for discrete-time or continuous-time dynamic graphs and group the models according to their architecture. We find that there are still graph types that are not or only rarely covered by existing GNN models. We point out where models are missing and give potential reasons for their absence. △ Less

Submitted 26 April, 2023; v1 submitted 6 April, 2022; originally announced April 2022.

Development of a Scale to Measure Technology Acceptance in Smart Agriculture

Authors: Rosemary J Thomas, Rebecca Whetton, Andy Doyle, David Coyle

Abstract: This paper describes the development of a scale to measure technology acceptance in smart agriculture. The scale is intended for use in diverse situations, ranging for the evaluation of existing technologies already in widespread use, to the evaluation of prototype systems. A systematic screening of prior literature was conducted to identify initial scale items regarding how technology acceptance… ▽ More This paper describes the development of a scale to measure technology acceptance in smart agriculture. The scale is intended for use in diverse situations, ranging for the evaluation of existing technologies already in widespread use, to the evaluation of prototype systems. A systematic screening of prior literature was conducted to identify initial scale items regarding how technology acceptance is currently understood and measured. These items were iteratively reviewed and systematically categorised to develop the final scale proposed in this paper. In future work, this scale will be validated through user studies. The purpose of this paper is to make the initial scale available to the research community with a view to initial use and further evaluation. △ Less

Submitted 29 March, 2022; originally announced March 2022.

BioSimulators: a central registry of simulation engines and services for recommending specific tools

Authors: Bilal Shaikh, Lucian P. Smith, Dan Vasilescu, Gnaneswara Marupilla, Michael Wilson, Eran Agmon, Henry Agnew, Steven S. Andrews, Azraf Anwar, Moritz E. Beber, Frank T. Bergmann, David Brooks, Lutz Brusch, Laurence Calzone, Kiri Choi, Joshua Cooper, John Detloff, Brian Drawert, Michel Dumontier, G. Bard Ermentrout, James R. Faeder, Andrew P. Freiburger, Fabian Fröhlich, Akira Funahashi, Alan Garny , et al. (46 additional authors not shown)

Abstract: Computational models have great potential to accelerate bioscience, bioengineering, and medicine. However, it remains challenging to reproduce and reuse simulations, in part, because the numerous formats and methods for simulating various subsystems and scales remain siloed by different software tools. For example, each tool must be executed through a distinct interface. To help investigators find… ▽ More Computational models have great potential to accelerate bioscience, bioengineering, and medicine. However, it remains challenging to reproduce and reuse simulations, in part, because the numerous formats and methods for simulating various subsystems and scales remain siloed by different software tools. For example, each tool must be executed through a distinct interface. To help investigators find and use simulation tools, we developed BioSimulators (https://biosimulators.org), a central registry of the capabilities of simulation tools and consistent Python, command-line, and containerized interfaces to each version of each tool. The foundation of BioSimulators is standards, such as CellML, SBML, SED-ML, and the COMBINE archive format, and validation tools for simulation projects and simulation tools that ensure these standards are used consistently. To help modelers find tools for particular projects, we have also used the registry to develop recommendation services. We anticipate that BioSimulators will help modelers exchange, reproduce, and combine simulations. △ Less

Submitted 13 March, 2022; originally announced March 2022.

Comments: 6 pages, 2 figures

Quantifying Daily Evolution of Mobile Software Based on Memory Allocator Churn

Authors: Gunnar Kudrjavets, Jeff Thomas, Aditya Kumar, Nachiappan Nagappan, Ayushi Rastogi

Abstract: The pace and volume of code churn necessary to evolve modern software systems present challenges for analyzing the performance impact of any set of code changes. Traditional methods used in performance analysis rely on extensive data collection and profiling, which often takes days. For large organizations utilizing Continuous Integration (CI) and Continuous Deployment (CD), these traditional tech… ▽ More The pace and volume of code churn necessary to evolve modern software systems present challenges for analyzing the performance impact of any set of code changes. Traditional methods used in performance analysis rely on extensive data collection and profiling, which often takes days. For large organizations utilizing Continuous Integration (CI) and Continuous Deployment (CD), these traditional techniques often fail to provide timely and actionable data. A different impact analysis method that allows for more efficient detection of performance regressions is needed. We propose the utilization of user mode memory allocator churn as a novel approach to performance engineering. User mode allocator churn acts as a proxy metric to evaluate the relative change in the cost of specific tasks. We prototyped the memory allocation churn methodology while engaged in performance engineering for a major iOS application. We find that calculating and analyzing memory allocator churn (a) results in deterministic measurements, (b) is efficient for determining the presence of both individual performance regressions and general performance-related trends, and (c) is a suitable alternative to measuring the task completion time. △ Less

Submitted 6 May, 2022; v1 submitted 8 March, 2022; originally announced March 2022.

Comments: 5 pages. To be published in Proceedings of The 9th International Conference on Mobile Software Engineering and Systems (MobileSoft '22). ACM, New York, NY, USA

Better Modelling Out-of-Distribution Regression on Distributed Acoustic Sensor Data Using Anchored Hidden State Mixup

Authors: Hasan Asyari Arief, Peter James Thomas, Tomasz Wiktorski

Abstract: Generalizing the application of machine learning models to situations where the statistical distribution of training and test data are different has been a complex problem. Our contributions in this paper are threefold: (1) we introduce an anchored-based Out of Distribution (OOD) Regression Mixup algorithm, leveraging manifold hidden state mixup and observation similarities to form a novel regular… ▽ More Generalizing the application of machine learning models to situations where the statistical distribution of training and test data are different has been a complex problem. Our contributions in this paper are threefold: (1) we introduce an anchored-based Out of Distribution (OOD) Regression Mixup algorithm, leveraging manifold hidden state mixup and observation similarities to form a novel regularization penalty, (2) we provide a first of its kind, high resolution Distributed Acoustic Sensor (DAS) dataset that is suitable for testing OOD regression modelling, allowing other researchers to benchmark progress in this area, and (3) we demonstrate with an extensive evaluation the generalization performance of the proposed method against existing approaches, then show that our method achieves state-of-the-art performance. Lastly, we also demonstrate a wider applicability of the proposed method by exhibiting improved generalization performances on other types of regression datasets, including Udacity and Rotation-MNIST datasets. △ Less

Submitted 22 February, 2022; originally announced February 2022.

Comments: TII Accepted Version

Journal ref: IEEE Transactions on Industrial Informatics (TII.2022.3154783)

Pragmatic Implementation of Reinforcement Algorithms For Path Finding On Raspberry Pi

Authors: Serena Raju, Sherin Shibu, Riya Mol Raji, Joel Thomas

Abstract: In this paper, pragmatic implementation of an indoor autonomous delivery system that exploits Reinforcement Learning algorithms for path planning and collision avoidance is audited. The proposed system is a cost-efficient approach that is implemented to facilitate a Raspberry Pi controlled four-wheel-drive non-holonomic robot map a grid. This approach computes and navigates the shortest path from… ▽ More In this paper, pragmatic implementation of an indoor autonomous delivery system that exploits Reinforcement Learning algorithms for path planning and collision avoidance is audited. The proposed system is a cost-efficient approach that is implemented to facilitate a Raspberry Pi controlled four-wheel-drive non-holonomic robot map a grid. This approach computes and navigates the shortest path from a source key point to a destination key point to carry out the desired delivery. Q learning and Deep-Q learning are used to find the optimal path while avoiding collision with static obstacles. This work defines an approach to deploy these two algorithms on a robot. A novel algorithm to decode an array of directions into accurate movements in a certain action space is also proposed. The procedure followed to dispatch this system with the said requirements is described, ergo presenting our proof of concept for indoor autonomous delivery vehicles. △ Less

Submitted 7 December, 2021; originally announced December 2021.

Comments: 5 pages, 7 figures

ACM Class: I.2.9; F.2.0

Enabling Reusable Physical Design Flows with Modular Flow Generators

Authors: Alex Carsello, James Thomas, Ankita Nayak, Po-Han Chen, Mark Horowitz, Priyanka Raina, Christopher Torng

Abstract: Achieving high code reuse in physical design flows is challenging but increasingly necessary to build complex systems. Unfortunately, existing approaches based on parameterized Tcl generators support very limited reuse and struggle to preserve reusable code as designers customize flows for specific designs and technologies. We present a vision and framework based on modular flow generators that en… ▽ More Achieving high code reuse in physical design flows is challenging but increasingly necessary to build complex systems. Unfortunately, existing approaches based on parameterized Tcl generators support very limited reuse and struggle to preserve reusable code as designers customize flows for specific designs and technologies. We present a vision and framework based on modular flow generators that encapsulates coarse-grain and fine-grain reusable code in modular nodes and assembles them into complete flows. The key feature is a flow consistency and instrumentation layer embedded in Python, which supports mechanisms for rapid and early feedback on inconsistent composition. The approach gradually types the Tcl language and allows both automatic and user-annotated static assertion checks. We evaluate the design flows of successive generations of silicon prototypes designed in TSMC16, TSMC28, TSMC40, SKY130, and IBM180 technologies, showing how our approach can enable significant code reuse in future flows. △ Less

Submitted 29 November, 2021; originally announced November 2021.

Multi-lingual agents through multi-headed neural networks

Authors: J. D. Thomas, R. Santos-Rodríguez, R. Piechocki, M. Anca

Abstract: This paper considers cooperative Multi-Agent Reinforcement Learning, focusing on emergent communication in settings where multiple pairs of independent learners interact at varying frequencies. In this context, multiple distinct and incompatible languages can emerge. When an agent encounters a speaker of an alternative language, there is a requirement for a period of adaptation before they can eff… ▽ More This paper considers cooperative Multi-Agent Reinforcement Learning, focusing on emergent communication in settings where multiple pairs of independent learners interact at varying frequencies. In this context, multiple distinct and incompatible languages can emerge. When an agent encounters a speaker of an alternative language, there is a requirement for a period of adaptation before they can efficiently converse. This adaptation results in the emergence of a new language and the forgetting of the previous language. In principle, this is an example of the Catastrophic Forgetting problem which can be mitigated by enabling the agents to learn and maintain multiple languages. We take inspiration from the Continual Learning literature and equip our agents with multi-headed neural networks which enable our agents to be multi-lingual. Our method is empirically validated within a referential MNIST based communication game and is shown to be able to maintain multiple languages where existing approaches cannot. △ Less

Submitted 22 November, 2021; originally announced November 2021.

Comments: Cooperative AI workshop NeurIPS 2021

RLOps: Development Life-cycle of Reinforcement Learning Aided Open RAN

Authors: Peizheng Li, Jonathan Thomas, Xiaoyang Wang, Ahmed Khalil, Abdelrahim Ahmad, Rui Inacio, Shipra Kapoor, Arjun Parekh, Angela Doufexi, Arman Shojaeifard, Robert Piechocki

Abstract: Radio access network (RAN) technologies continue to evolve, with Open RAN gaining the most recent momentum. In the O-RAN specifications, the RAN intelligent controllers (RICs) are software-defined orchestration and automation functions for the intelligent management of RAN. This article introduces principles for machine learning (ML), in particular, reinforcement learning (RL) applications in the… ▽ More Radio access network (RAN) technologies continue to evolve, with Open RAN gaining the most recent momentum. In the O-RAN specifications, the RAN intelligent controllers (RICs) are software-defined orchestration and automation functions for the intelligent management of RAN. This article introduces principles for machine learning (ML), in particular, reinforcement learning (RL) applications in the O-RAN stack. Furthermore, we review the state-of-the-art research in wireless networks and cast it onto the RAN framework and the hierarchy of the O-RAN architecture. We provide a taxonomy for the challenges faced by ML/RL models throughout the development life-cycle: from the system specification to production deployment (data acquisition, model design, testing and management, etc.). To address the challenges, we integrate a set of existing MLOps principles with unique characteristics when RL agents are considered. This paper discusses a systematic model development, testing and validation life-cycle, termed: RLOps. We discuss fundamental parts of RLOps, which include: model specification, development, production environment serving, operations monitoring and safety/security. Based on these principles, we propose the best practices for RLOps to achieve an automated and reproducible model development process. At last, a holistic data analytics platform rooted in the O-RAN deployment is designed and implemented, aiming to embrace and fulfil the aforementioned principles and best practices of RLOps. △ Less

Submitted 25 November, 2022; v1 submitted 12 November, 2021; originally announced November 2021.

Comments: 17 pages, 6 figrues

Journal ref: IEEE Access (2022), vol. 10, pp. 113808-113826

A Note on the Modeling Power of Different Graph Types

Authors: Josephine M. Thomas, Silvia Beddar-Wiesing, Alice Moallemy-Oureh, Rüdiger Nather

Abstract: Graphs can have different properties that lead to several graph types and may allow for a varying representation of diverse information. In order to clarify the modeling power of graphs, we introduce a partial order on the most common graph types based on an expressivity relation. The expressivity relation quantifies how many properties a graph type can encode compared to another type. Additionall… ▽ More Graphs can have different properties that lead to several graph types and may allow for a varying representation of diverse information. In order to clarify the modeling power of graphs, we introduce a partial order on the most common graph types based on an expressivity relation. The expressivity relation quantifies how many properties a graph type can encode compared to another type. Additionally, we show that all attributed graph types are equally expressive and have the same modeling power. △ Less

Submitted 7 September, 2022; v1 submitted 22 September, 2021; originally announced September 2021.

Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challenges

Authors: Bernd Bischl, Martin Binder, Michel Lang, Tobias Pielok, Jakob Richter, Stefan Coors, Janek Thomas, Theresa Ullmann, Marc Becker, Anne-Laure Boulesteix, Difan Deng, Marius Lindauer

Abstract: Most machine learning algorithms are configured by one or several hyperparameters that must be carefully chosen and often considerably impact performance. To avoid a time consuming and unreproducible manual trial-and-error process to find well-performing hyperparameter configurations, various automatic hyperparameter optimization (HPO) methods, e.g., based on resampling error estimation for superv… ▽ More Most machine learning algorithms are configured by one or several hyperparameters that must be carefully chosen and often considerably impact performance. To avoid a time consuming and unreproducible manual trial-and-error process to find well-performing hyperparameter configurations, various automatic hyperparameter optimization (HPO) methods, e.g., based on resampling error estimation for supervised machine learning, can be employed. After introducing HPO from a general perspective, this paper reviews important HPO methods such as grid or random search, evolutionary algorithms, Bayesian optimization, Hyperband and racing. It gives practical recommendations regarding important choices to be made when conducting HPO, including the HPO algorithms themselves, performance evaluation, how to combine HPO with ML pipelines, runtime improvements, and parallelization. This work is accompanied by an appendix that contains information on specific software packages in R and Python, as well as information and recommended hyperparameter search spaces for specific learning algorithms. We also provide notebooks that demonstrate concepts from this work as supplementary files. △ Less

Submitted 24 November, 2021; v1 submitted 13 July, 2021; originally announced July 2021.

Linear solvers for power grid optimization problems: a review of GPU-accelerated linear solvers

Authors: Kasia Swirydowicz, Eric Darve, Wesley Jones, Jonathan Maack, Shaked Regev, Michael A. Saunders, Stephen J. Thomas, Slaven Peles

Abstract: The linear equations that arise in interior methods for constrained optimization are sparse symmetric indefinite and become extremely ill-conditioned as the interior method converges. These linear systems present a challenge for existing solver frameworks based on sparse LU or LDL^T decompositions. We benchmark five well known direct linear solver packages using matrices extracted from power grid… ▽ More The linear equations that arise in interior methods for constrained optimization are sparse symmetric indefinite and become extremely ill-conditioned as the interior method converges. These linear systems present a challenge for existing solver frameworks based on sparse LU or LDL^T decompositions. We benchmark five well known direct linear solver packages using matrices extracted from power grid optimization problems. The achieved solution accuracy varies greatly among the packages. None of the tested packages delivers significant GPU acceleration for our test cases. △ Less

Submitted 13 August, 2021; v1 submitted 25 June, 2021; originally announced June 2021.