-
CNIMA: A Universal Evaluation Framework and Automated Approach for Assessing Second Language Dialogues
Authors:
Rena Gao,
Jingxuan Wu,
Carsten Roever,
Xuetong Wu,
Jing Wu,
Long Lv,
Jey Han Lau
Abstract:
We develop CNIMA (Chinese Non-Native Interactivity Measurement and Automation), a Chinese-as-a-second-language labelled dataset with 10K dialogues. We annotate CNIMA using an evaluation framework -- originally introduced for English-as-a-second-language dialogues -- that assesses micro-level features (e.g.\ backchannels) and macro-level interactivity labels (e.g.\ topic management) and test the fr…
▽ More
We develop CNIMA (Chinese Non-Native Interactivity Measurement and Automation), a Chinese-as-a-second-language labelled dataset with 10K dialogues. We annotate CNIMA using an evaluation framework -- originally introduced for English-as-a-second-language dialogues -- that assesses micro-level features (e.g.\ backchannels) and macro-level interactivity labels (e.g.\ topic management) and test the framework's transferability from English to Chinese. We found the framework robust across languages and revealed universal and language-specific relationships between micro-level and macro-level features. Next, we propose an approach to automate the evaluation and find strong performance, creating a new tool for automated second language assessment. Our system can be adapted to other languages easily as it uses large language models and as such does not require large-scale annotated training data.
△ Less
Submitted 29 August, 2024;
originally announced August 2024.
-
Interaction Matters: An Evaluation Framework for Interactive Dialogue Assessment on English Second Language Conversations
Authors:
Rena Gao,
Carsten Roever,
Jey Han Lau
Abstract:
We present an evaluation framework for interactive dialogue assessment in the context of English as a Second Language (ESL) speakers. Our framework collects dialogue-level interactivity labels (e.g., topic management; 4 labels in total) and micro-level span features (e.g., backchannels; 17 features in total). Given our annotated data, we study how the micro-level features influence the (higher lev…
▽ More
We present an evaluation framework for interactive dialogue assessment in the context of English as a Second Language (ESL) speakers. Our framework collects dialogue-level interactivity labels (e.g., topic management; 4 labels in total) and micro-level span features (e.g., backchannels; 17 features in total). Given our annotated data, we study how the micro-level features influence the (higher level) interactivity quality of ESL dialogues by constructing various machine learning-based models. Our results demonstrate that certain micro-level features strongly correlate with interactivity quality, like reference word (e.g., she, her, he), revealing new insights about the interaction between higher-level dialogue quality and lower-level linguistic signals. Our framework also provides a means to assess ESL communication, which is useful for language assessment.
△ Less
Submitted 8 July, 2024;
originally announced July 2024.
-
An Introduction to Software Ecosystems
Authors:
Tom Mens,
Coen De Roover
Abstract:
This chapter defines and presents different kinds of software ecosystems. The focus is on the development, tooling and analytics aspects of software ecosystems, i.e., communities of software developers and the interconnected software components (e.g., projects, libraries, packages, repositories, plug-ins, apps) they are developing and maintaining. The technical and social dependencies between thes…
▽ More
This chapter defines and presents different kinds of software ecosystems. The focus is on the development, tooling and analytics aspects of software ecosystems, i.e., communities of software developers and the interconnected software components (e.g., projects, libraries, packages, repositories, plug-ins, apps) they are developing and maintaining. The technical and social dependencies between these developers and software components form a socio-technical dependency network, and the dynamics of this network change over time. We classify and provide several examples of such ecosystems. The chapter also introduces and clarifies the relevant terms needed to understand and analyse these ecosystems, as well as the techniques and research methods that can be used to analyse different aspects of these ecosystems.
△ Less
Submitted 28 July, 2023;
originally announced July 2023.
-
Formalizing, Verifying and Applying ISA Security Guarantees as Universal Contracts
Authors:
Sander Huyghebaert,
Steven Keuchel,
Coen De Roover,
Dominique Devriese
Abstract:
Progress has recently been made on specifying instruction set architectures (ISAs) in executable formalisms rather than through prose. However, to date, those formal specifications are limited to the functional aspects of the ISA and do not cover its security guarantees. We present a novel, general method for formally specifying an ISAs security guarantees to (1) balance the needs of ISA implement…
▽ More
Progress has recently been made on specifying instruction set architectures (ISAs) in executable formalisms rather than through prose. However, to date, those formal specifications are limited to the functional aspects of the ISA and do not cover its security guarantees. We present a novel, general method for formally specifying an ISAs security guarantees to (1) balance the needs of ISA implementations (hardware) and clients (software), (2) can be semi-automatically verified to hold for the ISA operational semantics, producing a high-assurance mechanically-verifiable proof, and (3) support informal and formal reasoning about security-critical software in the presence of adversarial code. Our method leverages universal contracts: software contracts that express bounds on the authority of arbitrary untrusted code. Universal contracts can be kept agnostic of software abstractions, and strike the right balance between requiring sufficient detail for reasoning about software and preserving implementation freedom of ISA designers and CPU implementers. We semi-automatically verify universal contracts against Sail implementations of ISA semantics using our Katamaran tool; a semi-automatic separation logic verifier for Sail which produces machine-checked proofs for successfully verified contracts. We demonstrate the generality of our method by applying it to two ISAs that offer very different security primitives: (1) MinimalCaps: a custom-built capability machine ISA and (2) a (somewhat simplified) version of RISC-V with PMP. We verify a femtokernel using the security guarantee we have formalized for RISC-V with PMP.
△ Less
Submitted 8 January, 2024; v1 submitted 8 June, 2023;
originally announced June 2023.
-
Security Risks of Porting C Programs to WebAssembly
Authors:
Quentin Stiévenart,
Coen De Roover,
Mohammad Ghafari
Abstract:
WebAssembly is a compilation target for cross-platform applications that is increasingly being used. In this paper, we investigate whether one can transparently cross-compile C programs to WebAssembly, and if not, what impact porting can have on their security. We compile 17,802 programs that exhibit common vulnerabilities to 64-bit x86 and to WebAssembly binaries, and we observe that the executio…
▽ More
WebAssembly is a compilation target for cross-platform applications that is increasingly being used. In this paper, we investigate whether one can transparently cross-compile C programs to WebAssembly, and if not, what impact porting can have on their security. We compile 17,802 programs that exhibit common vulnerabilities to 64-bit x86 and to WebAssembly binaries, and we observe that the execution of 4,911 binaries produces different results across these platforms. Through manual inspection, we identify three classes of root causes for such differences: the use of a different standard library implementation, the lack of security measures in WebAssembly, and the different semantics of the execution environments. We describe our observations and discuss the ones that are critical from a security point of view and need most attention from developers. We conclude that compiling an existing C program to WebAssembly for cross-platform distribution may require source code adaptations; otherwise, the security of the WebAssembly application may be at risk.
△ Less
Submitted 22 December, 2021;
originally announced December 2021.
-
The Security Risk of Lacking Compiler Protection in WebAssembly
Authors:
Quentin Stiévenart,
Coen De Roover,
Mohammad Ghafari
Abstract:
WebAssembly is increasingly used as the compilation target for cross-platform applications. In this paper, we investigate whether one can rely on the security measures enforced by existing C compilers when compiling C programs to WebAssembly. We compiled 4,469 C programs with known buffer overflow vulnerabilities to x86 code and to WebAssembly, and observed the outcome of the execution of the gene…
▽ More
WebAssembly is increasingly used as the compilation target for cross-platform applications. In this paper, we investigate whether one can rely on the security measures enforced by existing C compilers when compiling C programs to WebAssembly. We compiled 4,469 C programs with known buffer overflow vulnerabilities to x86 code and to WebAssembly, and observed the outcome of the execution of the generated code to differ for 1,088 programs. Through manual inspection, we identified that the root cause for these is the lack of security measures such as stack canaries in the generated WebAssembly: while x86 code crashes upon a stack-based buffer overflow, the corresponding WebAssembly continues to be executed. We conclude that compiling an existing C program to WebAssembly without additional precautions may hamper its security, and we encourage more research in this direction.
△ Less
Submitted 2 November, 2021;
originally announced November 2021.
-
On the Impact of Security Vulnerabilities in the npm and RubyGems Dependency Networks
Authors:
Ahmed Zerouali,
Tom Mens,
Alexandre Decan,
Coen De Roover
Abstract:
The increasing interest in open source software has led to the emergence of large language-specific package distributions of reusable software libraries, such as npm and RubyGems. These software packages can be subject to vulnerabilities that may expose dependent packages through explicitly declared dependencies. Using Snyk's vulnerability database, this article empirically studies vulnerabilities…
▽ More
The increasing interest in open source software has led to the emergence of large language-specific package distributions of reusable software libraries, such as npm and RubyGems. These software packages can be subject to vulnerabilities that may expose dependent packages through explicitly declared dependencies. Using Snyk's vulnerability database, this article empirically studies vulnerabilities affecting npm and RubyGems packages. We analyse how and when these vulnerabilities are disclosed and fixed, and how their prevalence changes over time. We also analyse how vulnerable packages expose their direct and indirect dependents to vulnerabilities. We distinguish between two types of dependents: packages distributed via the package manager, and external GitHub projects depending on npm packages. We observe that the number of vulnerabilities in npm is increasing and being disclosed faster than vulnerabilities in RubyGems. For both package distributions, the time required to disclose vulnerabilities is increasing over time. Vulnerabilities in npm packages affect a median of 30 package releases, while this is 59 releases in RubyGems packages. A large proportion of external GitHub projects is exposed to vulnerabilities coming from direct or indirect dependencies. 33% and 40% of dependency vulnerabilities to which projects and packages are exposed, respectively, have their fixes in more recent releases within the same major release range of the used dependency. Our findings reveal that more effort is needed to better secure open source package distributions.
△ Less
Submitted 28 March, 2022; v1 submitted 12 June, 2021;
originally announced June 2021.
-
Prioritising Server Side Reachability via Inter-process Concolic Testing
Authors:
Maarten Vandercammen,
Laurent Christophe,
Dario Di Nucci,
Wolfgang De Meuter,
Coen De Roover
Abstract:
Context: Most approaches to automated white-box testing consider the client side and the server side of a web application in isolation from each other. Such testers lack a whole-program perspective on the web application under test.
Inquiry: We hypothesise that an additional whole-program perspective would enable the tester to discover which server side errors can be triggered by an actual end u…
▽ More
Context: Most approaches to automated white-box testing consider the client side and the server side of a web application in isolation from each other. Such testers lack a whole-program perspective on the web application under test.
Inquiry: We hypothesise that an additional whole-program perspective would enable the tester to discover which server side errors can be triggered by an actual end user accessing the application through the client, and which ones can only be triggered in hypothetical scenarios.
Approach: In this paper, we explore the idea of employing such a whole-program perspective in testing. To this end, we develop , a novel concolic tester which operates on full-stack JavaScript web applications, where both the client and the server side are JavaScript processes communicating via asynchronous messages -- as enabled by the WebSocket or Socket.IO-libraries.
Knowledge: We find that the whole-program perspective enables discerning high-priority errors, which are reachable from a particular client, from low-priority errors, which are not accessible through the tested client. Another benefit of the perspective is that it allows the automated tester to construct practical, step-by-step scenarios for triggering server side errors from the end user's perspective.
Grounding: We apply on a collection of web applications to evaluate how effective testing is in distinguishing between high- and low-priority errors. The results show that correctly classifies the majority of server errors.
Importance: This paper demonstrates the feasibility of testing as a novel approach for automatically testing web applications. Classifying errors as being of high or low importance aids developers in prioritising bugs that might be encountered by users, and postponing the diagnosis of bugs that are less easily reached.
△ Less
Submitted 4 March, 2021; v1 submitted 30 October, 2020;
originally announced October 2020.
-
Search-based Tier Assignment for Optimising Offline Availability in Multi-tier Web Applications
Authors:
Laure Philips,
Joeri De Koster,
Wolfgang De Meuter,
Coen De Roover
Abstract:
Web programmers are often faced with several challenges in the development process of modern, rich internet applications. Technologies for the different tiers of the application have to be selected: a server-side language, a combination of JavaScript, HTML and CSS for the client, and a database technology. Meeting the expectations of contemporary web applications requires even more effort from the…
▽ More
Web programmers are often faced with several challenges in the development process of modern, rich internet applications. Technologies for the different tiers of the application have to be selected: a server-side language, a combination of JavaScript, HTML and CSS for the client, and a database technology. Meeting the expectations of contemporary web applications requires even more effort from the developer: many state of the art libraries must be mastered and glued together. This leads to an impedance mismatch problem between the different technologies and it is up to the programmer to align them manually. Multi-tier or tierless programming is a web programming paradigm that provides one language for the different tiers of the web application, allowing the programmer to focus on the actual program logic instead of the accidental complexity that comes from combining several technologies. While current tierless approaches therefore relieve the burden of having to combine different technologies into one application, the distribution of the code is explicitly tied into the program. Certain distribution decisions have an impact on crosscutting concerns such as information security or offline availability. Moreover, adapting the programs such that the application complies better with these concerns often leads to code tangling, rendering the program more difficult to understand and maintain. We introduce an approach to multi-tier programming where the tierless code is decoupled from the tier specification. The developer implements the web application in terms of slices and an external specification that assigns the slices to tiers. A recommender system completes the picture for those slices that do not have a fixed placement and proposes slice refinements as well. This recommender system tries to optimise the tier specification with respect to one or more crosscutting concerns. This is in contrast with current cutting edge solutions that hide distribution decisions from the programmer. In this paper we show that slices, together with a recommender system, enable the developer to experiment with different placements of slices, until the distribution of the code satisfies the programmer's needs. We present a search-based recommender system that maximises the offline availability of a web application and a concrete implementation of these concepts in the tier-splitting tool Stip.js.
△ Less
Submitted 4 December, 2017;
originally announced December 2017.
-
Modeling Library Dependencies and Updates in Large Software Repository Universes
Authors:
Raula Gaikovina Kula,
Coen De Roover,
Daniel M. German,
Takashi Ishio,
Katsuro Inoue
Abstract:
Popular (re)use of third-party open-source software (OSS) is evidence of the impact of hosting repositories like maven on software development today. Updating libraries is crucial, with recent studies highlighting the associated vulnerabilities with aging OSS libraries. The decision to migrate to a newer library can range from trivial (security threat) to complex (assessment of work required to ac…
▽ More
Popular (re)use of third-party open-source software (OSS) is evidence of the impact of hosting repositories like maven on software development today. Updating libraries is crucial, with recent studies highlighting the associated vulnerabilities with aging OSS libraries. The decision to migrate to a newer library can range from trivial (security threat) to complex (assessment of work required to accommodate the changes). By leveraging the `wisdom of the software repository crowd' we propose a simple and efficient approach to recommending `consented' library updates. Our Software Universe Graph (SUG) models library dependency and update information mined from super repositories to provide different metrics and visualizations that aid in the update decision. To evaluate, we first constructed a SUG from 188,951 nodes of 6,374 maven unique artifacts. Then, we demonstrate how our metrics and visualizations are applied through real-world examples. As an extension, we show how the SUG can compare dependencies between different super repositories. From a sample of 100 GitHub applications, our method found that on average 79% similar overlapping dependencies combinations exist between the maven and github super repository universes.
△ Less
Submitted 14 September, 2017;
originally announced September 2017.
