-
Evaluation of the Programming Skills of Large Language Models
Authors:
Luc Bryan Heitz,
Joun Chamas,
Christopher Scherb
Abstract:
The advent of Large Language Models (LLM) has revolutionized the efficiency and speed with which tasks are completed, marking a significant leap in productivity through technological innovation. As these chatbots tackle increasingly complex tasks, the challenge of assessing the quality of their outputs has become paramount. This paper critically examines the output quality of two leading LLMs, Ope…
▽ More
The advent of Large Language Models (LLM) has revolutionized the efficiency and speed with which tasks are completed, marking a significant leap in productivity through technological innovation. As these chatbots tackle increasingly complex tasks, the challenge of assessing the quality of their outputs has become paramount. This paper critically examines the output quality of two leading LLMs, OpenAI's ChatGPT and Google's Gemini AI, by comparing the quality of programming code generated in both their free versions. Through the lens of a real-world example coupled with a systematic dataset, we investigate the code quality produced by these LLMs. Given their notable proficiency in code generation, this aspect of chatbot capability presents a particularly compelling area for analysis. Furthermore, the complexity of programming code often escalates to levels where its verification becomes a formidable task, underscoring the importance of our study. This research aims to shed light on the efficacy and reliability of LLMs in generating high-quality programming code, an endeavor that has significant implications for the field of software development and beyond.
△ Less
Submitted 23 May, 2024;
originally announced May 2024.
-
Divide, Conquer and Verify: Improving Symbolic Execution Performance
Authors:
Christopher Scherb,
Luc Bryan Heitz,
Hermann Grieder,
Olivier Mattmann
Abstract:
Symbolic Execution is a formal method that can be used to verify the behavior of computer programs and detect software vulnerabilities. Compared to other testing methods such as fuzzing, Symbolic Execution has the advantage of providing formal guarantees about the program. However, despite advances in performance in recent years, Symbolic Execution is too slow to be applied to real-world software.…
▽ More
Symbolic Execution is a formal method that can be used to verify the behavior of computer programs and detect software vulnerabilities. Compared to other testing methods such as fuzzing, Symbolic Execution has the advantage of providing formal guarantees about the program. However, despite advances in performance in recent years, Symbolic Execution is too slow to be applied to real-world software. This is primarily caused by the \emph{path explosion problem} as well as by the computational complexity of SMT solving. In this paper, we present a divide-and-conquer approach for symbolic execution by executing individual slices and later combining the side effects. This way, the overall problem size is kept small, reducing the impact of computational complexity on large problems.
△ Less
Submitted 7 November, 2023; v1 submitted 5 October, 2023;
originally announced October 2023.
-
CyMed: A Framework for Testing Cybersecurity of Connected Medical Devices
Authors:
Christopher Scherb,
Adrian Hadayah,
Luc Bryan Heitz
Abstract:
Connected Medical Devices (CMDs) have a large impact on patients as they allow them to lead a more normal life. Any malfunction could not only remove the health benefits the CMDs provide, they could also cause further harm to the patient. Due to this, there are many safety regulations which must be adhered to prior to a CMD entering the market. However, while many detailed safety regulations exist…
▽ More
Connected Medical Devices (CMDs) have a large impact on patients as they allow them to lead a more normal life. Any malfunction could not only remove the health benefits the CMDs provide, they could also cause further harm to the patient. Due to this, there are many safety regulations which must be adhered to prior to a CMD entering the market. However, while many detailed safety regulations exist, there are a fundamental lack of cybersecurity frameworks applicable to CMDs. While there are recent regulations which aim to enforce cybersecurity practices, they are vague and do not contain the concrete steps necessary to implement cybersecurity. This paper aims to fill that gap by describing a framework, CyMed, to be used by vendors and ens-users, which contains concrete measures to improve the resilience of CMDs against cyber attack. The CyMed framework is subsequently evaluated based on practical tests as well as expert interviews.
△ Less
Submitted 5 October, 2023;
originally announced October 2023.
-
An Adaptable Approach for Successful SIEM Adoption in Companies
Authors:
Maximilian Rosenberg,
Bettina Schneider,
Christopher Scherb,
Petra Maria Asprion
Abstract:
In corporations around the world, the topic of cybersecurity and information security is becoming increasingly important as the number of cyberattacks on themselves continues to grow. Nowadays, it is no longer just a matter of protecting against cyberattacks, but rather of detecting such attacks at an early stage and responding accordingly. There is currently no generic methodological approach for…
▽ More
In corporations around the world, the topic of cybersecurity and information security is becoming increasingly important as the number of cyberattacks on themselves continues to grow. Nowadays, it is no longer just a matter of protecting against cyberattacks, but rather of detecting such attacks at an early stage and responding accordingly. There is currently no generic methodological approach for the implementation of Security Information and Event Management (SIEM) systems that takes academic aspects into account and can be applied independently of the product or developers of the systems. Applying Hevner's design science research approach, the goal of this paper is to develop a holistic procedure model for implementing respective SIEM systems in corporations. According to the study during the validation phase, the procedure model was verified to be applicable. As desire for future research, the procedure model should be applied in various implementation projects in different enterprises to analyze its applicability and completeness.
△ Less
Submitted 2 August, 2023;
originally announced August 2023.
-
A Serious Game for Simulating Cyberattacks to Teach Cybersecurity
Authors:
Christopher Scherb,
Luc Bryan Heitz,
Frank Grimberg,
Hermann Grieder,
Marcel Maurer
Abstract:
With the rising number of cyberattacks, such as ransomware attacks and cyber espionage, educating non-cybersecurity professionals to recognize threats has become more important than ever before. However, traditional training methods, such as phishing awareness campaigns, training videos and assessments have proven to be less effective over time. Therefore, it is time to rethink the approach on how…
▽ More
With the rising number of cyberattacks, such as ransomware attacks and cyber espionage, educating non-cybersecurity professionals to recognize threats has become more important than ever before. However, traditional training methods, such as phishing awareness campaigns, training videos and assessments have proven to be less effective over time. Therefore, it is time to rethink the approach on how to train cyber awareness. In this paper we suggest an alternative approach -- a serious game -- to educate awareness for common cyberattacks. While many serious games for cybersecurity education exist, all follow a very similar approach: showing people the effects of a cyber attack on their own system or company network. For example, one of the main tasks in these games is to sort out phishing mails. We developed and evaluated a new type of cybersecurity game: an attack simulator, which shows the entire setting from a different perspective. Instead of sorting out phishing mails the players should write phishing mails to trick potential victims and use other forms of cyberattacks. Our game explains the intention of each attack and shows the consequences of a successful attack. This way, we hope, players will get a better understanding on how to detect cyberattacks.
△ Less
Submitted 4 May, 2023;
originally announced May 2023.
-
Tangle Centric Networking (TCN)
Authors:
Christopher Scherb,
Dennis Grewe,
Christian Tschudin
Abstract:
Today's Internet is heavily used for multimedia streaming from cloud backends, while the Internet of Things (IoT) reverses the traditional data flow, with high data volumes produced at the network edge. Information Centric Networking (ICN) advocates against a host-centric communication model which is promising for distributed edge computing environments and the execution of IoT applications in a d…
▽ More
Today's Internet is heavily used for multimedia streaming from cloud backends, while the Internet of Things (IoT) reverses the traditional data flow, with high data volumes produced at the network edge. Information Centric Networking (ICN) advocates against a host-centric communication model which is promising for distributed edge computing environments and the execution of IoT applications in a decentralized fashion. By using naming schemes, data is tightly coupled to names instead of hosts which simplifies discovery and access to data and services. However, the tight coupling challenges network performance due to additional synchronization overhead of large data volumes and services. We present Tangle Centric Networking (TCN) -- a decentralized data structure for coordinated distributed applications and data exchange following principles of ICN. TCN can react on data and service changes and update them accordingly in network nodes, provide distributed data structures and enable cooperative work on the same data without huge overhead by using Tangles for coordination. We implemented TCN in simulations and evaluated the concept against a base line scenario.
△ Less
Submitted 15 August, 2021;
originally announced August 2021.
-
SwarMS: Swarm-Computations for Mobile Scenarios
Authors:
Christopher Scherb,
Pascal Bürklin,
Christian Tschudin
Abstract:
Nowadays, most network systems are based on fixed and reliable infrastructure. In this context Information Centric Networking (ICN) is a novel network approach, where data is in the focus instead of hosts. Therefore, requests for data are independent from the location where the data is actually stored on. This property is optimal for infrastructure-less and swarm networking. However, the ICN does…
▽ More
Nowadays, most network systems are based on fixed and reliable infrastructure. In this context Information Centric Networking (ICN) is a novel network approach, where data is in the focus instead of hosts. Therefore, requests for data are independent from the location where the data is actually stored on. This property is optimal for infrastructure-less and swarm networking. However, the ICN does not provide support for networks without infrastructure. In this paper we present SwarMS, an architecture for swarm on-boarding, swarm coordination and swarm computations in ICN style networks. We use append-only logs to solve the challenges of loose mobile swarm organisation and executing computations. By replicating tasks on multiple nodes we achieve more reliability and trust in results.
△ Less
Submitted 15 August, 2021;
originally announced August 2021.
