-
SoK: Prudent Evaluation Practices for Fuzzing
Authors:
Moritz Schloegel,
Nils Bars,
Nico Schiller,
Lukas Bernhard,
Tobias Scharnowski,
Addison Crump,
Arash Ale Ebrahim,
Nicolai Bissantz,
Marius Muench,
Thorsten Holz
Abstract:
Fuzzing has proven to be a highly effective approach to uncover software bugs over the past decade. After AFL popularized the groundbreaking concept of lightweight coverage feedback, the field of fuzzing has seen a vast amount of scientific work proposing new techniques, improving methodological aspects of existing strategies, or porting existing methods to new domains. All such work must demonstr…
▽ More
Fuzzing has proven to be a highly effective approach to uncover software bugs over the past decade. After AFL popularized the groundbreaking concept of lightweight coverage feedback, the field of fuzzing has seen a vast amount of scientific work proposing new techniques, improving methodological aspects of existing strategies, or porting existing methods to new domains. All such work must demonstrate its merit by showing its applicability to a problem, measuring its performance, and often showing its superiority over existing works in a thorough, empirical evaluation. Yet, fuzzing is highly sensitive to its target, environment, and circumstances, e.g., randomness in the testing process. After all, relying on randomness is one of the core principles of fuzzing, governing many aspects of a fuzzer's behavior. Combined with the often highly difficult to control environment, the reproducibility of experiments is a crucial concern and requires a prudent evaluation setup. To address these threats to validity, several works, most notably Evaluating Fuzz Testing by Klees et al., have outlined how a carefully designed evaluation setup should be implemented, but it remains unknown to what extent their recommendations have been adopted in practice. In this work, we systematically analyze the evaluation of 150 fuzzing papers published at the top venues between 2018 and 2023. We study how existing guidelines are implemented and observe potential shortcomings and pitfalls. We find a surprising disregard of the existing guidelines regarding statistical tests and systematic errors in fuzzing evaluations. For example, when investigating reported bugs, ...
△ Less
Submitted 16 May, 2024;
originally announced May 2024.
-
History-Independent Concurrent Objects
Authors:
Hagit Attiya,
Michael A. Bender,
Martin Farach-Colton,
Rotem Oshman,
Noa Schiller
Abstract:
A data structure is called history independent if its internal memory representation does not reveal the history of operations applied to it, only its current state. In this paper we study history independence for concurrent data structures, and establish foundational possibility and impossibility results. We show that a large class of concurrent objects cannot be implemented from smaller base obj…
▽ More
A data structure is called history independent if its internal memory representation does not reveal the history of operations applied to it, only its current state. In this paper we study history independence for concurrent data structures, and establish foundational possibility and impossibility results. We show that a large class of concurrent objects cannot be implemented from smaller base objects in a manner that is both wait-free and history independent; but if we settle for either lock-freedom instead of wait-freedom or for a weak notion of history independence, then at least one object in the class, multi-valued single-reader single-writer registers, can be implemented from smaller base objects, binary registers.
On the other hand, using large base objects, we give a strong possibility result in the form of a universal construction: an object with $s$ possible states can be implemented in a wait-free, history-independent manner from compare-and-swap base objects that each have $O(s + 2^n)$ possible memory states, where $n$ is the number of processes in the system.
△ Less
Submitted 21 March, 2024;
originally announced March 2024.
-
Asynchronous Fully-Decentralized SGD in the Cluster-Based Model
Authors:
Hagit Attiya,
Noa Schiller
Abstract:
This paper presents fault-tolerant asynchronous Stochastic Gradient Descent (SGD) algorithms. SGD is widely used for approximating the minimum of a cost function $Q$, as a core part of optimization and learning algorithms. Our algorithms are designed for the cluster-based model, which combines message-passing and shared-memory communication layers. Processes may fail by crashing, and the algorithm…
▽ More
This paper presents fault-tolerant asynchronous Stochastic Gradient Descent (SGD) algorithms. SGD is widely used for approximating the minimum of a cost function $Q$, as a core part of optimization and learning algorithms. Our algorithms are designed for the cluster-based model, which combines message-passing and shared-memory communication layers. Processes may fail by crashing, and the algorithm inside each cluster is wait-free, using only reads and writes.
For a strongly convex function $Q$, our algorithm tolerates any number of failures, and provides convergence rate that yields the maximal distributed acceleration over the optimal convergence rate of sequential SGD.
For arbitrary functions, the convergence rate has an additional term that depends on the maximal difference between the parameters at the same iteration. (This holds under standard assumptions on $Q$.) In this case, the algorithm obtains the same convergence rate as sequential SGD, up to a logarithmic factor. This is achieved by using, at each iteration, a multidimensional approximate agreement algorithm, tailored for the cluster-based model.
The algorithm for arbitrary functions requires that at least a majority of the clusters contain at least one nonfaulty process. We prove that this condition is necessary when optimizing some non-convex functions.
△ Less
Submitted 13 June, 2023; v1 submitted 22 February, 2022;
originally announced February 2022.
-
Cross-linguistic differences in gender congruency effects: Evidence from meta-analyses
Authors:
Audrey Bürki,
Emiel van den Hoven,
Niels O. Schiller,
Nikolay DImitrov
Abstract:
It has been proposed that the order in which words are prepared for production depends on the speaker's language. When producing the translation equivalent of the small cat, speakers of German or Dutch select the gender-marked determiner at a relatively early stage of production. Speakers of French or Italian postpone the encoding of a determiner or adjective until the phonological form of the nou…
▽ More
It has been proposed that the order in which words are prepared for production depends on the speaker's language. When producing the translation equivalent of the small cat, speakers of German or Dutch select the gender-marked determiner at a relatively early stage of production. Speakers of French or Italian postpone the encoding of a determiner or adjective until the phonological form of the noun is available. Hence, even though the words are produced in the same order (e.g., die kleine Katze in German, le petit chat in French), they are not planned in the same order and might require different amounts of advanced planning prior to production onset. This distinction between early and late selection languages was proposed to account for the observation that speakers of Germanic and Slavic languages, but not of Romance languages, are slower to name pictures in the context of a distractor word of a different gender. Meta-analyses are conducted to provide the first direct test of this cross-linguistic difference and to test a prediction of the late selection hypothesis. They confirm the existence of the gender congruency effect in German/Slavic languages and its absence in Romance languages when target and distractor words are presented simultaneously. They do not allow confirming the hypothesis that in the latter languages, a similar effect emerges when the presentation of the distractor is delayed. Overall, these analyses confirm the cross-linguistic difference but show that the evidence available to date is not sufficient to confirm or reject the late selection hypothesis as an explanation of this difference. We highlight specific directions for future research.
△ Less
Submitted 26 April, 2023; v1 submitted 8 September, 2021;
originally announced September 2021.
-
Optimal Resilience in Systems that Mix Shared Memory and Message Passing
Authors:
Hagit Attiya,
Sweta Kumari,
Noa Schiller
Abstract:
We investigate the minimal number of failures that can partition a system where processes communicate both through shared memory and by message passing. We prove that this number precisely captures the resilience that can be achieved by algorithms that implement a variety of shared objects, like registers and atomic snapshots, and solve common tasks, like randomized consensus, approximate agreemen…
▽ More
We investigate the minimal number of failures that can partition a system where processes communicate both through shared memory and by message passing. We prove that this number precisely captures the resilience that can be achieved by algorithms that implement a variety of shared objects, like registers and atomic snapshots, and solve common tasks, like randomized consensus, approximate agreement and renaming. This has implications for the m&m-model and for the hybrid, cluster-based model.
△ Less
Submitted 19 December, 2020;
originally announced December 2020.
