Showing 1–32 of 32 results for author: Schneider, F

M$\mathbf5$ -- A Diverse Benchmark to Assess the Performance of Large Multimodal Models Across Multilingual and Multicultural Vision-Language Tasks

Authors: Florian Schneider, Sunayana Sitaram

Abstract: Since the release of ChatGPT, the field of Natural Language Processing has experienced rapid advancements, particularly in Large Language Models (LLMs) and their multimodal counterparts, Large Multimodal Models (LMMs). Despite their impressive capabilities, LLMs often exhibit significant performance disparities across different languages and cultural contexts, as demonstrated by various text-only… ▽ More Since the release of ChatGPT, the field of Natural Language Processing has experienced rapid advancements, particularly in Large Language Models (LLMs) and their multimodal counterparts, Large Multimodal Models (LMMs). Despite their impressive capabilities, LLMs often exhibit significant performance disparities across different languages and cultural contexts, as demonstrated by various text-only benchmarks. However, current research lacks such benchmarks for multimodal visio-linguistic settings. This work fills this gap by introducing M5, the first comprehensive benchmark designed to evaluate LMMs on diverse vision-language tasks within a multilingual and multicultural context. M5 includes eight datasets covering five tasks and $41$ languages, with a focus on underrepresented languages and culturally diverse images. Furthermore, we introduce two novel datasets, M5-VGR and M5-VLOD, including a new Visio-Linguistic Outlier Detection task, in which all evaluated open-source models fail to significantly surpass the random baseline. Through extensive evaluation and analyses, we highlight substantial task-agnostic performance disparities between high- and low-resource languages. Moreover, we show that larger models do not necessarily outperform smaller ones in a multilingual setting. △ Less

Submitted 4 July, 2024; originally announced July 2024.

Why do LLaVA Vision-Language Models Reply to Images in English?

Authors: Musashi Hinck, Carolin Holtermann, Matthew Lyle Olson, Florian Schneider, Sungduk Yu, Anahita Bhiwandiwalla, Anne Lauscher, Shaoyen Tseng, Vasudev Lal

Abstract: We uncover a surprising multilingual bias occurring in a popular class of multimodal vision-language models (VLMs). Including an image in the query to a LLaVA-style VLM significantly increases the likelihood of the model returning an English response, regardless of the language of the query. This paper investigates the causes of this loss with a two-pronged approach that combines extensive ablatio… ▽ More We uncover a surprising multilingual bias occurring in a popular class of multimodal vision-language models (VLMs). Including an image in the query to a LLaVA-style VLM significantly increases the likelihood of the model returning an English response, regardless of the language of the query. This paper investigates the causes of this loss with a two-pronged approach that combines extensive ablation of the design space with a mechanistic analysis of the models' internal representations of image and text inputs. Both approaches indicate that the issue stems in the language modelling component of the LLaVA model. Statistically, we find that switching the language backbone for a bilingual language model has the strongest effect on reducing this error. Mechanistically, we provide compelling evidence that visual inputs are not mapped to a similar space as text ones, and that intervening on intermediary attention layers can reduce this bias. Our findings provide important insights to researchers and engineers seeking to understand the crossover between multimodal and multilingual spaces, and contribute to the goal of developing capable and inclusive VLMs for non-English contexts. △ Less

Submitted 2 July, 2024; originally announced July 2024.

Comments: Pre-print

Reducing the cost of posterior sampling in linear inverse problems via task-dependent score learning

Authors: Fabian Schneider, Duc-Lam Duong, Matti Lassas, Maarten V. de Hoop, Tapio Helin

Abstract: Score-based diffusion models (SDMs) offer a flexible approach to sample from the posterior distribution in a variety of Bayesian inverse problems. In the literature, the prior score is utilized to sample from the posterior by different methods that require multiple evaluations of the forward mapping in order to generate a single posterior sample. These methods are often designed with the objective… ▽ More Score-based diffusion models (SDMs) offer a flexible approach to sample from the posterior distribution in a variety of Bayesian inverse problems. In the literature, the prior score is utilized to sample from the posterior by different methods that require multiple evaluations of the forward mapping in order to generate a single posterior sample. These methods are often designed with the objective of enabling the direct use of the unconditional prior score and, therefore, task-independent training. In this paper, we focus on linear inverse problems, when evaluation of the forward mapping is computationally expensive and frequent posterior sampling is required for new measurement data, such as in medical imaging. We demonstrate that the evaluation of the forward mapping can be entirely bypassed during posterior sample generation. Instead, without introducing any error, the computational effort can be shifted to an offline task of training the score of a specific diffusion-like random process. In particular, the training is task-dependent requiring information about the forward mapping but not about the measurement data. It is shown that the conditional score corresponding to the posterior can be obtained from the auxiliary score by suitable affine transformations. We prove that this observation generalizes to the framework of infinite-dimensional diffusion models introduced recently and provide numerical analysis of the method. Moreover, we validate our findings with numerical experiments. △ Less

Submitted 24 May, 2024; originally announced May 2024.

Comments: 23 pages, 2 figues

MSC Class: 62F15; 65N21; 68Q32; 60Hxx; 60Jxx

Learn to Code Sustainably: An Empirical Study on LLM-based Green Code Generation

Authors: Tina Vartziotis, Ippolyti Dellatolas, George Dasoulas, Maximilian Schmidt, Florian Schneider, Tim Hoffmann, Sotirios Kotsopoulos, Michael Keckeisen

Abstract: The increasing use of information technology has led to a significant share of energy consumption and carbon emissions from data centers. These contributions are expected to rise with the growing demand for big data analytics, increasing digitization, and the development of large artificial intelligence (AI) models. The need to address the environmental impact of software development has led to in… ▽ More The increasing use of information technology has led to a significant share of energy consumption and carbon emissions from data centers. These contributions are expected to rise with the growing demand for big data analytics, increasing digitization, and the development of large artificial intelligence (AI) models. The need to address the environmental impact of software development has led to increased interest in green (sustainable) coding and claims that the use of AI models can lead to energy efficiency gains. Here, we provide an empirical study on green code and an overview of green coding practices, as well as metrics used to quantify the sustainability awareness of AI models. In this framework, we evaluate the sustainability of auto-generated code. The auto-generate codes considered in this study are produced by generative commercial AI language models, GitHub Copilot, OpenAI ChatGPT-3, and Amazon CodeWhisperer. Within our methodology, in order to quantify the sustainability awareness of these AI models, we propose a definition of the code's "green capacity", based on certain sustainability metrics. We compare the performance and green capacity of human-generated code and code generated by the three AI language models in response to easy-to-hard problem statements. Our findings shed light on the current capacity of AI models to contribute to sustainable software development. △ Less

Submitted 5 March, 2024; originally announced March 2024.

Towards RehabCoach: Design and Preliminary Evaluation of a Conversational Agent Supporting Unsupervised Therapy after Stroke

Authors: Giada Devittori, Mehdi Akeddar, Alexandra Retevoi, Fabian Schneider, Viktoria Cvetkova, Daria Dinacci, Antonella Califfi, Paolo Rossi, Claudio Petrillo, Tobias Kowatsch, Olivier Lambercy

Abstract: Unsupervised therapy after stroke is a promising way to boost therapy dose without significantly increasing the workload on healthcare professionals. However, it raises important challenges, such as lower adherence to therapy in the absence of social interaction with therapists. We present the initial prototype of RehabCoach, a novel smartphone-based app with conversational agent to support unsupe… ▽ More Unsupervised therapy after stroke is a promising way to boost therapy dose without significantly increasing the workload on healthcare professionals. However, it raises important challenges, such as lower adherence to therapy in the absence of social interaction with therapists. We present the initial prototype of RehabCoach, a novel smartphone-based app with conversational agent to support unsupervised therapy. RehabCoach is designed to increase patients engagement and adherence to therapy and to provide information (e.g., about stroke, health) in an interactive and user-friendly manner. We report on the design and usability evaluation of the first prototype of RehabCoach, assessed by four stroke patients and five healthcare professionals, who interacted with the app in a single testing session. Task completion time and success rates were measured for 15 representative tasks, and participants assessed usability via questionnaires and a semi-structured interview. Results show that it was feasible for stroke patients to successfully interact with RehabCoach (task success $\geq$ 93 $\%$) without requiring extensive training. Participants positively rated the usability of RehabCoach (mean mHealth App Usability Questionnaire score: 1.3 for primary users, 1.4 for healthcare professionals, on a scale from 1 (positive evaluation) to 7). The feedback collected in this work opens the door to further enhance RehabCoach as an interactive digital tool to support unsupervised rehabilitation. △ Less

Submitted 2 March, 2024; originally announced March 2024.

Comments: 6 pages, 3 figures

Kronecker-Factored Approximate Curvature for Modern Neural Network Architectures

Authors: Runa Eschenhagen, Alexander Immer, Richard E. Turner, Frank Schneider, Philipp Hennig

Abstract: The core components of many modern neural network architectures, such as transformers, convolutional, or graph neural networks, can be expressed as linear layers with $\textit{weight-sharing}$. Kronecker-Factored Approximate Curvature (K-FAC), a second-order optimisation method, has shown promise to speed up neural network training and thereby reduce computational costs. However, there is currentl… ▽ More The core components of many modern neural network architectures, such as transformers, convolutional, or graph neural networks, can be expressed as linear layers with $\textit{weight-sharing}$. Kronecker-Factored Approximate Curvature (K-FAC), a second-order optimisation method, has shown promise to speed up neural network training and thereby reduce computational costs. However, there is currently no framework to apply it to generic architectures, specifically ones with linear weight-sharing layers. In this work, we identify two different settings of linear weight-sharing layers which motivate two flavours of K-FAC -- $\textit{expand}$ and $\textit{reduce}$. We show that they are exact for deep linear networks with weight-sharing in their respective setting. Notably, K-FAC-reduce is generally faster than K-FAC-expand, which we leverage to speed up automatic hyperparameter selection via optimising the marginal likelihood for a Wide ResNet. Finally, we observe little difference between these two K-FAC variations when using them to train both a graph neural network and a vision transformer. However, both variations are able to reach a fixed validation metric target in $50$-$75\%$ of the number of steps of a first-order reference run, which translates into a comparable improvement in wall-clock time. This highlights the potential of applying K-FAC to modern neural network architectures. △ Less

Submitted 11 January, 2024; v1 submitted 1 November, 2023; originally announced November 2023.

Comments: NeurIPS 2023

Accelerating Generalized Linear Models by Trading off Computation for Uncertainty

Authors: Lukas Tatzel, Jonathan Wenger, Frank Schneider, Philipp Hennig

Abstract: Bayesian Generalized Linear Models (GLMs) define a flexible probabilistic framework to model categorical, ordinal and continuous data, and are widely used in practice. However, exact inference in GLMs is prohibitively expensive for large datasets, thus requiring approximations in practice. The resulting approximation error adversely impacts the reliability of the model and is not accounted for in… ▽ More Bayesian Generalized Linear Models (GLMs) define a flexible probabilistic framework to model categorical, ordinal and continuous data, and are widely used in practice. However, exact inference in GLMs is prohibitively expensive for large datasets, thus requiring approximations in practice. The resulting approximation error adversely impacts the reliability of the model and is not accounted for in the uncertainty of the prediction. In this work, we introduce a family of iterative methods that explicitly model this error. They are uniquely suited to parallel modern computing hardware, efficiently recycle computations, and compress information to reduce both the time and memory requirements for GLMs. As we demonstrate on a realistically large classification problem, our method significantly accelerates training compared to competitive baselines by trading off reduced computation for increased uncertainty. △ Less

Submitted 7 February, 2024; v1 submitted 31 October, 2023; originally announced October 2023.

Comments: Main text: 11 pages, 6 figures; Supplements: 13 pages, 2 figures

Improved Scheduling with a Shared Resource

Authors: Christoph Damerius, Peter Kling, Florian Schneider

Abstract: We consider the following shared-resource scheduling problem: Given a set of jobs $J$, for each $j\in J$ we must schedule a job-specific processing volume of $v_j>0$. A total resource of $1$ is available at any time. Jobs have a resource requirement $r_j\in[0,1]$, and the resources assigned to them may vary over time. However, assigning them less will cause a proportional slowdown. We consider t… ▽ More We consider the following shared-resource scheduling problem: Given a set of jobs $J$, for each $j\in J$ we must schedule a job-specific processing volume of $v_j>0$. A total resource of $1$ is available at any time. Jobs have a resource requirement $r_j\in[0,1]$, and the resources assigned to them may vary over time. However, assigning them less will cause a proportional slowdown. We consider two settings. In the first, we seek to minimize the makespan in an online setting: The resource assignment of a job must be fixed before the next job arrives. Here we give an optimal $e/(e-1)$-competitive algorithm with runtime $\mathcal{O}(n\cdot \log n)$. In the second, we aim to minimize the total completion time. We use a continuous linear programming (CLP) formulation for the fractional total completion time and combine it with a previously known dominance property from malleable job scheduling to obtain a lower bound on the total completion time. We extract structural properties by considering a geometrical representation of a CLP's primal-dual pair. We combine the CLP schedule with a greedy schedule to obtain a $(3/2+\varepsilon)$-approximation for this setting. This improves upon the so far best-known approximation factor of $2$. △ Less

Submitted 10 October, 2023; v1 submitted 9 October, 2023; originally announced October 2023.

Comments: Submitted to COCOA 2023, Full Version

Benchmarking Neural Network Training Algorithms

Authors: George E. Dahl, Frank Schneider, Zachary Nado, Naman Agarwal, Chandramouli Shama Sastry, Philipp Hennig, Sourabh Medapati, Runa Eschenhagen, Priya Kasimbeg, Daniel Suo, Juhan Bae, Justin Gilmer, Abel L. Peirson, Bilal Khan, Rohan Anil, Mike Rabbat, Shankar Krishnan, Daniel Snider, Ehsan Amid, Kongtao Chen, Chris J. Maddison, Rakshith Vasudev, Michal Badura, Ankush Garg, Peter Mattson

Abstract: Training algorithms, broadly construed, are an essential part of every deep learning pipeline. Training algorithm improvements that speed up training across a wide variety of workloads (e.g., better update rules, tuning protocols, learning rate schedules, or data selection schemes) could save time, save computational resources, and lead to better, more accurate, models. Unfortunately, as a communi… ▽ More Training algorithms, broadly construed, are an essential part of every deep learning pipeline. Training algorithm improvements that speed up training across a wide variety of workloads (e.g., better update rules, tuning protocols, learning rate schedules, or data selection schemes) could save time, save computational resources, and lead to better, more accurate, models. Unfortunately, as a community, we are currently unable to reliably identify training algorithm improvements, or even determine the state-of-the-art training algorithm. In this work, using concrete experiments, we argue that real progress in speeding up training requires new benchmarks that resolve three basic challenges faced by empirical comparisons of training algorithms: (1) how to decide when training is complete and precisely measure training time, (2) how to handle the sensitivity of measurements to exact workload details, and (3) how to fairly compare algorithms that require hyperparameter tuning. In order to address these challenges, we introduce a new, competitive, time-to-result benchmark using multiple workloads running on fixed hardware, the AlgoPerf: Training Algorithms benchmark. Our benchmark includes a set of workload variants that make it possible to detect benchmark submissions that are more robust to workload changes than current widely-used methods. Finally, we evaluate baseline submissions constructed using various optimizers that represent current practice, as well as other optimizers that have recently received attention in the literature. These baseline results collectively demonstrate the feasibility of our benchmark, show that non-trivial gaps between methods exist, and set a provisional state-of-the-art for future benchmark submissions to try and surpass. △ Less

Submitted 12 June, 2023; originally announced June 2023.

Comments: 102 pages, 8 figures, 41 tables

Moving Target Defense for Service-oriented Mission-critical Networks

Authors: Doğanalp Ergenç, Florian Schneider, Peter Kling, Mathias Fischer

Abstract: Modern mission-critical systems (MCS) are increasingly softwarized and interconnected. As a result, their complexity increased, and so their vulnerability against cyber-attacks. The current adoption of virtualization and service-oriented architectures (SOA) in MCSs provides additional flexibility that can be leveraged to withstand and mitigate attacks, e.g., by moving critical services or data flo… ▽ More Modern mission-critical systems (MCS) are increasingly softwarized and interconnected. As a result, their complexity increased, and so their vulnerability against cyber-attacks. The current adoption of virtualization and service-oriented architectures (SOA) in MCSs provides additional flexibility that can be leveraged to withstand and mitigate attacks, e.g., by moving critical services or data flows. This enables the deployment of strategies for moving target defense (MTD), which allows stripping attackers of their asymmetric advantage from the long reconnaissance of MCSs. However, it is challenging to design MTD strategies, given the diverse threat landscape, resource limitations, and potential degradation in service availability. In this paper, we combine two optimization models to explore feasible service configurations for SOA-based systems and to derive subsequent MTD actions with their time schedule based on an attacker-defender game. Our results indicate that even for challenging and diverse attack scenarios, our models can defend the system by up to 90% of the system operation time with a limited MTD defender budget. △ Less

Submitted 17 March, 2023; originally announced March 2023.

ArchiSound: Audio Generation with Diffusion

Authors: Flavio Schneider

Abstract: The recent surge in popularity of diffusion models for image generation has brought new attention to the potential of these models in other areas of media generation. One area that has yet to be fully explored is the application of diffusion models to audio generation. Audio generation requires an understanding of multiple aspects, such as the temporal dimension, long term structure, multiple laye… ▽ More The recent surge in popularity of diffusion models for image generation has brought new attention to the potential of these models in other areas of media generation. One area that has yet to be fully explored is the application of diffusion models to audio generation. Audio generation requires an understanding of multiple aspects, such as the temporal dimension, long term structure, multiple layers of overlapping sounds, and the nuances that only trained listeners can detect. In this work, we investigate the potential of diffusion models for audio generation. We propose a set of models to tackle multiple aspects, including a new method for text-conditional latent audio diffusion with stacked 1D U-Nets, that can generate multiple minutes of music from a textual description. For each model, we make an effort to maintain reasonable inference speed, targeting real-time on a single consumer GPU. In addition to trained models, we provide a collection of open source libraries with the hope of simplifying future work in the field. Samples can be found at https://bit.ly/audio-diffusion. Codes are at https://github.com/archinetai/audio-diffusion-pytorch. △ Less

Submitted 30 January, 2023; originally announced January 2023.

Comments: Master Thesis at ETH Zurich

Moûsai: Text-to-Music Generation with Long-Context Latent Diffusion

Authors: Flavio Schneider, Ojasv Kamal, Zhijing Jin, Bernhard Schölkopf

Abstract: Recent years have seen the rapid development of large generative models for text; however, much less research has explored the connection between text and another "language" of communication -- music. Music, much like text, can convey emotions, stories, and ideas, and has its own unique structure and syntax. In our work, we bridge text and music via a text-to-music generation model that is highly… ▽ More Recent years have seen the rapid development of large generative models for text; however, much less research has explored the connection between text and another "language" of communication -- music. Music, much like text, can convey emotions, stories, and ideas, and has its own unique structure and syntax. In our work, we bridge text and music via a text-to-music generation model that is highly efficient, expressive, and can handle long-term structure. Specifically, we develop Moûsai, a cascading two-stage latent diffusion model that can generate multiple minutes of high-quality stereo music at 48kHz from textual descriptions. Moreover, our model features high efficiency, which enables real-time inference on a single consumer GPU with a reasonable speed. Through experiments and property analyses, we show our model's competence over a variety of criteria compared with existing music generation models. Lastly, to promote the open-source culture, we provide a collection of open-source libraries with the hope of facilitating future work in the field. We open-source the following: Codes: https://github.com/archinetai/audio-diffusion-pytorch; music samples for this paper: http://bit.ly/44ozWDH; all music samples for all models: https://bit.ly/audio-diffusion. △ Less

Submitted 23 October, 2023; v1 submitted 27 January, 2023; originally announced January 2023.

Intrinsic Dimension for Large-Scale Geometric Learning

Authors: Maximilian Stubbemann, Tom Hanika, Friedrich Martin Schneider

Abstract: The concept of dimension is essential to grasp the complexity of data. A naive approach to determine the dimension of a dataset is based on the number of attributes. More sophisticated methods derive a notion of intrinsic dimension (ID) that employs more complex feature functions, e.g., distances between data points. Yet, many of these approaches are based on empirical observations, cannot cope wi… ▽ More The concept of dimension is essential to grasp the complexity of data. A naive approach to determine the dimension of a dataset is based on the number of attributes. More sophisticated methods derive a notion of intrinsic dimension (ID) that employs more complex feature functions, e.g., distances between data points. Yet, many of these approaches are based on empirical observations, cannot cope with the geometric character of contemporary datasets, and do lack an axiomatic foundation. A different approach was proposed by V. Pestov, who links the intrinsic dimension axiomatically to the mathematical concentration of measure phenomenon. First methods to compute this and related notions for ID were computationally intractable for large-scale real-world datasets. In the present work, we derive a computationally feasible method for determining said axiomatic ID functions. Moreover, we demonstrate how the geometric properties of complex data are accounted for in our modeling. In particular, we propose a principle way to incorporate neighborhood information, as in graph data, into the ID. This allows for new insights into common graph learning procedures, which we illustrate by experiments on the Open Graph Benchmark. △ Less

Submitted 17 April, 2023; v1 submitted 11 October, 2022; originally announced October 2022.

Comments: 18 pages, 4 tables, 3 figures. This is the version accepted to TMLR, see: https://openreview.net/forum?id=85BfDdYMBY

Journal ref: Transactions on Machine Learning Research, 2023

An Application of Farkas' Lemma to Finite-Valued Constraint Satisfaction Problems over Infinite Domains

Authors: Friedrich Martin Schneider, Caterina Viola

Abstract: We show a universal algebraic local characterisation of the expressive power of finite-valued languages with domains of arbitrary cardinality and containing arbitrary many cost functions. We show a universal algebraic local characterisation of the expressive power of finite-valued languages with domains of arbitrary cardinality and containing arbitrary many cost functions. △ Less

Submitted 10 August, 2022; v1 submitted 9 August, 2022; originally announced August 2022.

Comments: 18 pages. The paper is based on a chapter from Caterina Viola's doctoral dissertation. This is a preprint of a manuscript accepted for publication in Journal of Mathematical Analysis and Applications (JMAA)

MSC Class: 46Axx ACM Class: G.0; F.2.0

Journal ref: J. Math. Anal. Appl. 517 (2023) 126591

Sparse Bayesian Learning for Complex-Valued Rational Approximations

Authors: Felix Schneider, Iason Papaioannou, Gerhard Müller

Abstract: Surrogate models are used to alleviate the computational burden in engineering tasks, which require the repeated evaluation of computationally demanding models of physical systems, such as the efficient propagation of uncertainties. For models that show a strongly non-linear dependence on their input parameters, standard surrogate techniques, such as polynomial chaos expansion, are not sufficient… ▽ More Surrogate models are used to alleviate the computational burden in engineering tasks, which require the repeated evaluation of computationally demanding models of physical systems, such as the efficient propagation of uncertainties. For models that show a strongly non-linear dependence on their input parameters, standard surrogate techniques, such as polynomial chaos expansion, are not sufficient to obtain an accurate representation of the original model response. Through applying a rational approximation instead, the approximation error can be efficiently reduced for models whose non-linearity is accurately described through a rational function. Specifically, our aim is to approximate complex-valued models. A common approach to obtain the coefficients in the surrogate is to minimize the sample-based error between model and surrogate in the least-square sense. In order to obtain an accurate representation of the original model and to avoid overfitting, the sample set has be two to three times the number of polynomial terms in the expansion. For models that require a high polynomial degree or are high-dimensional in terms of their input parameters, this number often exceeds the affordable computational cost. To overcome this issue, we apply a sparse Bayesian learning approach to the rational approximation. Through a specific prior distribution structure, sparsity is induced in the coefficients of the surrogate model. The denominator polynomial coefficients as well as the hyperparameters of the problem are determined through a type-II-maximum likelihood approach. We apply a quasi-Newton gradient-descent algorithm in order to find the optimal denominator coefficients and derive the required gradients through application of $\mathbb{CR}$-calculus. △ Less

Submitted 27 September, 2022; v1 submitted 6 June, 2022; originally announced June 2022.

Comments: 27 pages, 13 figures

Solving integer multi-objective optimization problems using TOPSIS, Differential Evolution and Tabu Search

Authors: Renato A. Krohling, Erick R. F. A. Schneider

Abstract: This paper presents a method to solve non-linear integer multiobjective optimization problems. First the problem is formulated using the Technique for Order Preference by Similarity to Ideal Solution (TOPSIS). Next, the Differential Evolution (DE) algorithm in its three versions (standard DE, DE best and DEGL) are used as optimizer. Since the solutions found by the DE algorithms are continuous, th… ▽ More This paper presents a method to solve non-linear integer multiobjective optimization problems. First the problem is formulated using the Technique for Order Preference by Similarity to Ideal Solution (TOPSIS). Next, the Differential Evolution (DE) algorithm in its three versions (standard DE, DE best and DEGL) are used as optimizer. Since the solutions found by the DE algorithms are continuous, the Tabu Search (TS) algorithm is employed to find integer solutions during the optimization process. Experimental results show the effectiveness of the proposed method. △ Less

Submitted 5 April, 2022; originally announced April 2022.

Comments: 8 pages

On Greedily Packing Anchored Rectangles

Authors: Christoph Damerius, Dominik Kaaser, Peter Kling, Florian Schneider

Abstract: Consider a set P of points in the unit square U, one of them being the origin. For each point p in P you may draw a rectangle in U with its lower-left corner in p. What is the maximum area such rectangles can cover without overlapping each other? Freedman [1969] posed this problem in 1969, asking whether one can always cover at least 50% of U. Over 40 years later, Dumitrescu and Tóth [2011] achiev… ▽ More Consider a set P of points in the unit square U, one of them being the origin. For each point p in P you may draw a rectangle in U with its lower-left corner in p. What is the maximum area such rectangles can cover without overlapping each other? Freedman [1969] posed this problem in 1969, asking whether one can always cover at least 50% of U. Over 40 years later, Dumitrescu and Tóth [2011] achieved the first constant coverage of 9.1%; since then, no significant progress was made. While 9.1% might seem low, the authors could not find any instance where their algorithm covers less than 50%, nourishing the hope to eventually prove a 50% bound. While we indeed significantly raise the algorithm's coverage to 39%, we extinguish the hope of reaching 50% by giving points for which the coverage is below 43.3%. Our analysis studies the algorithm's average and worst-case density of so-called tiles, which represent the area where a given point can freely choose its maximum-area rectangle. Our approachis comparatively general and may potentially help in analyzing related algorithms. △ Less

Submitted 16 February, 2021; originally announced February 2021.

Cockpit: A Practical Debugging Tool for the Training of Deep Neural Networks

Authors: Frank Schneider, Felix Dangel, Philipp Hennig

Abstract: When engineers train deep learning models, they are very much 'flying blind'. Commonly used methods for real-time training diagnostics, such as monitoring the train/test loss, are limited. Assessing a network's training process solely through these performance indicators is akin to debugging software without access to internal states through a debugger. To address this, we present Cockpit, a colle… ▽ More When engineers train deep learning models, they are very much 'flying blind'. Commonly used methods for real-time training diagnostics, such as monitoring the train/test loss, are limited. Assessing a network's training process solely through these performance indicators is akin to debugging software without access to internal states through a debugger. To address this, we present Cockpit, a collection of instruments that enable a closer look into the inner workings of a learning machine, and a more informative and meaningful status report for practitioners. It facilitates the identification of learning phases and failure modes, like ill-chosen hyperparameters. These instruments leverage novel higher-order information about the gradient distribution and curvature, which has only recently become efficiently accessible. We believe that such a debugging tool, which we open-source for PyTorch, is a valuable help in troubleshooting the training process. By revealing new insights, it also more generally contributes to explainability and interpretability of deep nets. △ Less

Submitted 26 October, 2021; v1 submitted 12 February, 2021; originally announced February 2021.

Comments: (NeurIPS 2021) Main text: 13 pages, 6 figures, 1 table; Supplements: 23 pages, 13 figures, 1 table, 1 listing

A Research Ecosystem for Secure Computing

Authors: Nadya Bliss, Lawrence A. Gordon, Daniel Lopresti, Fred Schneider, Suresh Venkatasubramanian

Abstract: Computing devices are vital to all areas of modern life and permeate every aspect of our society. The ubiquity of computing and our reliance on it has been accelerated and amplified by the COVID-19 pandemic. From education to work environments to healthcare to defense to entertainment - it is hard to imagine a segment of modern life that is not touched by computing. The security of computers, syst… ▽ More Computing devices are vital to all areas of modern life and permeate every aspect of our society. The ubiquity of computing and our reliance on it has been accelerated and amplified by the COVID-19 pandemic. From education to work environments to healthcare to defense to entertainment - it is hard to imagine a segment of modern life that is not touched by computing. The security of computers, systems, and applications has been an active area of research in computer science for decades. However, with the confluence of both the scale of interconnected systems and increased adoption of artificial intelligence, there are many research challenges the community must face so that our society can continue to benefit and risks are minimized, not multiplied. Those challenges range from security and trust of the information ecosystem to adversarial artificial intelligence and machine learning. Along with basic research challenges, more often than not, securing a system happens after the design or even deployment, meaning the security community is routinely playing catch-up and attempting to patch vulnerabilities that could be exploited any minute. While security measures such as encryption and authentication have been widely adopted, questions of security tend to be secondary to application capability. There needs to be a sea-change in the way we approach this critically important aspect of the problem: new incentives and education are at the core of this change. Now is the time to refocus research community efforts on developing interconnected technologies with security "baked in by design" and creating an ecosystem that ensures adoption of promising research developments. To realize this vision, two additional elements of the ecosystem are necessary - proper incentive structures for adoption and an educated citizenry that is well versed in vulnerabilities and risks. △ Less

Submitted 4 January, 2021; originally announced January 2021.

Comments: A Computing Community Consortium (CCC) white paper, 5 pages

Report number: ccc2020whitepaper_13

Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers

Authors: Robin M. Schmidt, Frank Schneider, Philipp Hennig

Abstract: Choosing the optimizer is considered to be among the most crucial design decisions in deep learning, and it is not an easy one. The growing literature now lists hundreds of optimization methods. In the absence of clear theoretical guidance and conclusive empirical evidence, the decision is often made based on anecdotes. In this work, we aim to replace these anecdotes, if not with a conclusive rank… ▽ More Choosing the optimizer is considered to be among the most crucial design decisions in deep learning, and it is not an easy one. The growing literature now lists hundreds of optimization methods. In the absence of clear theoretical guidance and conclusive empirical evidence, the decision is often made based on anecdotes. In this work, we aim to replace these anecdotes, if not with a conclusive ranking, then at least with evidence-backed heuristics. To do so, we perform an extensive, standardized benchmark of fifteen particularly popular deep learning optimizers while giving a concise overview of the wide range of possible choices. Analyzing more than $50,000$ individual runs, we contribute the following three points: (i) Optimizer performance varies greatly across tasks. (ii) We observe that evaluating multiple optimizers with default parameters works approximately as well as tuning the hyperparameters of a single, fixed optimizer. (iii) While we cannot discern an optimization method clearly dominating across all tested tasks, we identify a significantly reduced subset of specific optimizers and parameter choices that generally lead to competitive results in our experiments: Adam remains a strong contender, with newer methods failing to significantly and consistently outperform it. Our open-sourced results are available as challenging and well-tuned baselines for more meaningful evaluations of novel optimization methods without requiring any further computational efforts. △ Less

Submitted 10 August, 2021; v1 submitted 3 July, 2020; originally announced July 2020.

Comments: Raw results: https://github.com/SirRob1997/Crowded-Valley---Results

ELITR Non-Native Speech Translation at IWSLT 2020

Authors: Dominik Macháček, Jonáš Kratochvíl, Sangeet Sagar, Matúš Žilinec, Ondřej Bojar, Thai-Son Nguyen, Felix Schneider, Philip Williams, Yuekun Yao

Abstract: This paper is an ELITR system submission for the non-native speech translation task at IWSLT 2020. We describe systems for offline ASR, real-time ASR, and our cascaded approach to offline SLT and real-time SLT. We select our primary candidates from a pool of pre-existing systems, develop a new end-to-end general ASR system, and a hybrid ASR trained on non-native speech. The provided small validati… ▽ More This paper is an ELITR system submission for the non-native speech translation task at IWSLT 2020. We describe systems for offline ASR, real-time ASR, and our cascaded approach to offline SLT and real-time SLT. We select our primary candidates from a pool of pre-existing systems, develop a new end-to-end general ASR system, and a hybrid ASR trained on non-native speech. The provided small validation set prevents us from carrying out a complex validation, but we submit all the unselected candidates for contrastive evaluation on the test set. △ Less

Submitted 5 June, 2020; originally announced June 2020.

Comments: IWSLT 2020

Meta-learning curiosity algorithms

Authors: Ferran Alet, Martin F. Schneider, Tomas Lozano-Perez, Leslie Pack Kaelbling

Abstract: We hypothesize that curiosity is a mechanism found by evolution that encourages meaningful exploration early in an agent's life in order to expose it to experiences that enable it to obtain high rewards over the course of its lifetime. We formulate the problem of generating curious behavior as one of meta-learning: an outer loop will search over a space of curiosity mechanisms that dynamically ada… ▽ More We hypothesize that curiosity is a mechanism found by evolution that encourages meaningful exploration early in an agent's life in order to expose it to experiences that enable it to obtain high rewards over the course of its lifetime. We formulate the problem of generating curious behavior as one of meta-learning: an outer loop will search over a space of curiosity mechanisms that dynamically adapt the agent's reward signal, and an inner loop will perform standard reinforcement learning using the adapted reward signal. However, current meta-RL methods based on transferring neural network weights have only generalized between very similar tasks. To broaden the generalization, we instead propose to meta-learn algorithms: pieces of code similar to those designed by humans in ML papers. Our rich language of programs combines neural networks with other building blocks such as buffers, nearest-neighbor modules and custom loss functions. We demonstrate the effectiveness of the approach empirically, finding two novel curiosity algorithms that perform on par or better than human-designed published curiosity algorithms in domains as disparate as grid navigation with image inputs, acrobot, lunar lander, ant and hopper. △ Less

Submitted 11 March, 2020; originally announced March 2020.

Comments: Published in ICLR 2020

Assessing the Search and Rescue Domain as an Applied and Realistic Benchmark for Robotic Systems

Authors: Frank E. Schneider, Dennis Wildermuth

Abstract: Aim of this paper is to provide a review of the state of the art in Search and Rescue (SAR) robotics. Suitable robotic applications in the SAR domain are described, and SAR-specific demands and requirements on the various components of a robotic system are pictured. Current research and development in SAR robotics is outlined, and an overview of robotic systems and sub-systems currently in use in… ▽ More Aim of this paper is to provide a review of the state of the art in Search and Rescue (SAR) robotics. Suitable robotic applications in the SAR domain are described, and SAR-specific demands and requirements on the various components of a robotic system are pictured. Current research and development in SAR robotics is outlined, and an overview of robotic systems and sub-systems currently in use in SAR and disaster response scenarios is given. Finally we show a number of possible research directions for SAR robots, which might change the overall design and operation of SAR robotics in the longer-term future. All this is meant to support our main idea of taking SAR applications as an applied benchmark for the Field Robotics (FR) domain. △ Less

Submitted 10 December, 2019; originally announced December 2019.

Journal ref: 2016 17th International Carpathian Control Conference (ICCC)

DeepOBS: A Deep Learning Optimizer Benchmark Suite

Authors: Frank Schneider, Lukas Balles, Philipp Hennig

Abstract: Because the choice and tuning of the optimizer affects the speed, and ultimately the performance of deep learning, there is significant past and recent research in this area. Yet, perhaps surprisingly, there is no generally agreed-upon protocol for the quantitative and reproducible evaluation of optimization strategies for deep learning. We suggest routines and benchmarks for stochastic optimizati… ▽ More Because the choice and tuning of the optimizer affects the speed, and ultimately the performance of deep learning, there is significant past and recent research in this area. Yet, perhaps surprisingly, there is no generally agreed-upon protocol for the quantitative and reproducible evaluation of optimization strategies for deep learning. We suggest routines and benchmarks for stochastic optimization, with special focus on the unique aspects of deep learning, such as stochasticity, tunability and generalization. As the primary contribution, we present DeepOBS, a Python package of deep learning optimization benchmarks. The package addresses key challenges in the quantitative assessment of stochastic optimizers, and automates most steps of benchmarking. The library includes a wide and extensible set of ready-to-use realistic optimization problems, such as training Residual Networks for image classification on ImageNet or character-level language prediction models, as well as popular classics like MNIST and CIFAR-10. The package also provides realistic baseline results for the most popular optimizers on these test problems, ensuring a fair comparison to the competition when benchmarking new optimizers, and without having to run costly experiments. It comes with output back-ends that directly produce LaTeX code for inclusion in academic publications. It supports TensorFlow and is available open source. △ Less

Submitted 13 March, 2019; originally announced March 2019.

Comments: Accepted at ICLR 2019. 9 pages, 3 figures, 2 tables

NFV and SDN - Key Technology Enablers for 5G Networks

Authors: Faqir Zarrar Yousaf, Michael Bredel, Sibylle Schaller, Fabian Schneider

Abstract: Communication networks are undergoing their next evolutionary step towards 5G. The 5G networks are envisioned to provide a flexible, scalable, agile and programmable network platform over which different services with varying requirements can be deployed and managed within strict performance bounds. In order to address these challenges a paradigm shift is taking place in the technologies that driv… ▽ More Communication networks are undergoing their next evolutionary step towards 5G. The 5G networks are envisioned to provide a flexible, scalable, agile and programmable network platform over which different services with varying requirements can be deployed and managed within strict performance bounds. In order to address these challenges a paradigm shift is taking place in the technologies that drive the networks, and thus their architecture. Innovative concepts and techniques are being developed to power the next generation mobile networks. At the heart of this development lie Network Function Virtualization and Software Defined Networking technologies, which are now recognized as being two of the key technology enablers for realizing 5G networks, and which have introduced a major change in the way network services are deployed and operated. For interested readers that are new to the field of SDN and NFV this paper provides an overview of both these technologies with reference to the 5G networks. Most importantly it describes how the two technologies complement each other and how they are expected to drive the networks of near future. △ Less

Submitted 19 June, 2018; originally announced June 2018.

Comments: This is an accepted version and consists of 11 pages, 9 figures and 32 references

Journal ref: F. Z. Yousaf, M. Bredel, S. Schaller and F. Schneider, "NFV and SDN - Key Technology Enablers for 5G Networks," in IEEE Journal on Selected Areas in Communications, vol. 35, no. 11, pp. 2468 - 2478, Nov. 2017

Intrinsic dimension and its application to association rules

Authors: Tom Hanika, Friedrich Martin Schneider, Gerd Stumme

Abstract: The curse of dimensionality in the realm of association rules is twofold. Firstly, we have the well known exponential increase in computational complexity with increasing item set size. Secondly, there is a \emph{related curse} concerned with the distribution of (spare) data itself in high dimension. The former problem is often coped with by projection, i.e., feature selection, whereas the best kn… ▽ More The curse of dimensionality in the realm of association rules is twofold. Firstly, we have the well known exponential increase in computational complexity with increasing item set size. Secondly, there is a \emph{related curse} concerned with the distribution of (spare) data itself in high dimension. The former problem is often coped with by projection, i.e., feature selection, whereas the best known strategy for the latter is avoidance. This work summarizes the first attempt to provide a computationally feasible method for measuring the extent of dimension curse present in a data set with respect to a particular class machine of learning procedures. This recent development enables the application of various other methods from geometric analysis to be investigated and applied in machine learning procedures in the presence of high dimension. △ Less

Submitted 15 May, 2018; originally announced May 2018.

Comments: 4 pages, 1 figure

MSC Class: 68T01 68T05 ACM Class: I.2.6

Intrinsic Dimension of Geometric Data Sets

Authors: Tom Hanika, Friedrich Martin Schneider, Gerd Stumme

Abstract: The curse of dimensionality is a phenomenon frequently observed in machine learning (ML) and knowledge discovery (KD). There is a large body of literature investigating its origin and impact, using methods from mathematics as well as from computer science. Among the mathematical insights into data dimensionality, there is an intimate link between the dimension curse and the phenomenon of measure c… ▽ More The curse of dimensionality is a phenomenon frequently observed in machine learning (ML) and knowledge discovery (KD). There is a large body of literature investigating its origin and impact, using methods from mathematics as well as from computer science. Among the mathematical insights into data dimensionality, there is an intimate link between the dimension curse and the phenomenon of measure concentration, which makes the former accessible to methods of geometric analysis. The present work provides a comprehensive study of the intrinsic geometry of a data set, based on Gromov's metric measure geometry and Pestov's axiomatic approach to intrinsic dimension. In detail, we define a concept of geometric data set and introduce a metric as well as a partial order on the set of isomorphism classes of such data sets. Based on these objects, we propose and investigate an axiomatic approach to the intrinsic dimension of geometric data sets and establish a concrete dimension function with the desired properties. Our model for data sets and their intrinsic dimension is computationally feasible and, moreover, adaptable to specific ML/KD-algorithms, as illustrated by various experiments. △ Less

Submitted 26 October, 2020; v1 submitted 24 January, 2018; originally announced January 2018.

Comments: v3: 33 pages, 3 figures, 2 tables

MSC Class: 03G10 51F99 68P05 68T01 ACM Class: I.2.6

Journal ref: Tohoku Math. J. (2) 74 (2022) 23-52

MacWilliams' extension theorem for infinite rings

Authors: Friedrich Martin Schneider, Jens Zumbrägel

Abstract: Finite Frobenius rings have been characterized as precisely those finite rings satisfying the MacWilliams extension property, by work of Wood. In the present note we offer a generalization of this remarkable result to the realm of Artinian rings. Namely, we prove that a left Artinian ring has the left MacWilliams property if and only if it is left pseudo-injective and its finitary left socle embed… ▽ More Finite Frobenius rings have been characterized as precisely those finite rings satisfying the MacWilliams extension property, by work of Wood. In the present note we offer a generalization of this remarkable result to the realm of Artinian rings. Namely, we prove that a left Artinian ring has the left MacWilliams property if and only if it is left pseudo-injective and its finitary left socle embeds into the semisimple quotient. Providing a topological perspective on the MacWilliams property, we also show that the finitary left socle of a left Artinian ring embeds into the semisimple quotient if and only if it admits a finitarily left torsion-free character, if and only if the Pontryagin dual of the regular left module is almost monothetic. In conclusion, an Artinian ring has the MacWilliams property if and only if it is finitarily Frobenius, i.e., it is quasi-Frobenius and its finitary socle embeds into the semisimple quotient. △ Less

Submitted 8 August, 2018; v1 submitted 18 September, 2017; originally announced September 2017.

Comments: 14 pages. To appear in Proceedings of the AMS

MSC Class: 16L60; 16P20; 94B05

Journal ref: Proc. Amer. Math. Soc. 147 (2019), 947-961

Evolution of Directed Triangle Motifs in the Google+ OSN

Authors: Doris Schiöberg, Fabian Schneider, Stefan Schmid, Steve Uhlig, Anja Feldmann

Abstract: Motifs are a fundamental building block and distinguishing feature of networks. While characteristic motif distribution have been found in many networks, very little is known today about the evolution of network motifs. This paper studies the most important motifs in social networks, triangles, and how directed triangle motifs change over time. Our chosen subject is one of the largest Online Socia… ▽ More Motifs are a fundamental building block and distinguishing feature of networks. While characteristic motif distribution have been found in many networks, very little is known today about the evolution of network motifs. This paper studies the most important motifs in social networks, triangles, and how directed triangle motifs change over time. Our chosen subject is one of the largest Online Social Networks, Google+. Google+ has two distinguishing features that make it particularly interesting: (1) it is a directed network, which yields a rich set of triangle motifs, and (2) it is a young and fast evolving network, whose role in the OSN space is still not fully understood. For the purpose of this study, we crawled the network over a time period of six weeks, collecting several snapshots. We find that some triangle types display significant dynamics, e.g., for some specific initial types, up to 20% of the instances evolve to other types. Due to the fast growth of the OSN in the observed time period, many new triangles emerge. We also observe that many triangles evolve into less-connected motifs (with less edges), suggesting that growth also comes with pruning. We complement the topological study by also considering publicly available user profile data (mostly geographic locations). The corresponding results shed some light on the semantics of the triangle motifs. Indeed, we find that users in more symmetric triangle motifs live closer together, indicating more personal relationships. In contrast, asymmetric links in motifs often point to faraway users with a high in-degree (celebrities). △ Less

Submitted 17 February, 2015; v1 submitted 15 February, 2015; originally announced February 2015.

Vive la Différence: Paxos vs. Viewstamped Replication vs. Zab

Authors: Robbert Van Renesse, Nicolas Schiper, Fred B. Schneider

Abstract: Paxos, Viewstamped Replication, and Zab are replication protocols that ensure high-availability in asynchronous environments with crash failures. Various claims have been made about similarities and differences between these protocols. But how does one determine whether two protocols are the same, and if not, how significant the differences are? We propose to address these questions using refine… ▽ More Paxos, Viewstamped Replication, and Zab are replication protocols that ensure high-availability in asynchronous environments with crash failures. Various claims have been made about similarities and differences between these protocols. But how does one determine whether two protocols are the same, and if not, how significant the differences are? We propose to address these questions using refinement mappings, where protocols are expressed as succinct specifications that are progressively refined to executable implementations. Doing so enables a principled understanding of the correctness of the different design decisions that went into implementing the various protocols. Additionally, it allowed us to identify key differences that have a significant impact on performance. △ Less

Submitted 27 February, 2014; v1 submitted 22 September, 2013; originally announced September 2013.

Comments: 16 pages

Revisiting Content Availability in Distributed Online Social Networks

Authors: Doris Schiöberg, Fabian Schneider, Gilles Tredan, Steve Uhlig, Anja Feldmann

Abstract: Online Social Networks (OSN) are among the most popular applications in today's Internet. Decentralized online social networks (DOSNs), a special class of OSNs, promise better privacy and autonomy than traditional centralized OSNs. However, ensuring availability of content when the content owner is not online remains a major challenge. In this paper, we rely on the structure of the social graphs u… ▽ More Online Social Networks (OSN) are among the most popular applications in today's Internet. Decentralized online social networks (DOSNs), a special class of OSNs, promise better privacy and autonomy than traditional centralized OSNs. However, ensuring availability of content when the content owner is not online remains a major challenge. In this paper, we rely on the structure of the social graphs underlying DOSN for replication. In particular, we propose that friends, who are anyhow interested in the content, are used to replicate the users content. We study the availability of such natural replication schemes via both theoretical analysis as well as simulations based on data from OSN users. We find that the availability of the content increases drastically when compared to the online time of the user, e. g., by a factor of more than 2 for 90% of the users. Thus, with these simple schemes we provide a baseline for any more complicated content replication scheme. △ Less

Submitted 4 October, 2012; originally announced October 2012.

Comments: 11pages, 12 figures; Technical report at TU Berlin, Department of Electrical Engineering and Computer Science (ISSN 1436-9915)

Report number: TU Berlin/EECS TR No. 2012 - 05

Nerio: Leader Election and Edict Ordering

Authors: Robbert van Renesse, Fred B. Schneider, Johannes Gehrke

Abstract: Coordination in a distributed system is facilitated if there is a unique process, the leader, to manage the other processes. The leader creates edicts and sends them to other processes for execution or forwarding to other processes. The leader may fail, and when this occurs a leader election protocol selects a replacement. This paper describes Nerio, a class of such leader election protocols. Coordination in a distributed system is facilitated if there is a unique process, the leader, to manage the other processes. The leader creates edicts and sends them to other processes for execution or forwarding to other processes. The leader may fail, and when this occurs a leader election protocol selects a replacement. This paper describes Nerio, a class of such leader election protocols. △ Less

Submitted 26 September, 2011; v1 submitted 23 September, 2011; originally announced September 2011.