-
Towards Using Multiple Iterated, Reproduced, and Replicated Experiments with Robots (MIRRER) for Evaluation and Benchmarking
Authors:
Adam Norton,
Brian Flynn
Abstract:
The robotics research field lacks formalized definitions and frameworks for evaluating advanced capabilities including generalizability (the ability for robots to perform tasks under varied contexts) and reproducibility (the performance of a reproduced robot capability in different labs under the same experimental conditions). This paper presents an initial conceptual framework, MIRRER, that unite…
▽ More
The robotics research field lacks formalized definitions and frameworks for evaluating advanced capabilities including generalizability (the ability for robots to perform tasks under varied contexts) and reproducibility (the performance of a reproduced robot capability in different labs under the same experimental conditions). This paper presents an initial conceptual framework, MIRRER, that unites the concepts of performance evaluation, benchmarking, and reproduced/replicated experimentation in order to facilitate comparable robotics research. Several open issues with the application of the framework are also presented.
△ Less
Submitted 8 August, 2024;
originally announced August 2024.
-
Rapid Biomedical Research Classification: The Pandemic PACT Advanced Categorisation Engine
Authors:
Omid Rohanian,
Mohammadmahdi Nouriborji,
Olena Seminog,
Rodrigo Furst,
Thomas Mendy,
Shanthi Levanita,
Zaharat Kadri-Alabi,
Nusrat Jabin,
Daniela Toale,
Georgina Humphreys,
Emilia Antonio,
Adrian Bucher,
Alice Norton,
David A. Clifton
Abstract:
This paper introduces the Pandemic PACT Advanced Categorisation Engine (PPACE) along with its associated dataset. PPACE is a fine-tuned model developed to automatically classify research abstracts from funded biomedical projects according to WHO-aligned research priorities. This task is crucial for monitoring research trends and identifying gaps in global health preparedness and response. Our appr…
▽ More
This paper introduces the Pandemic PACT Advanced Categorisation Engine (PPACE) along with its associated dataset. PPACE is a fine-tuned model developed to automatically classify research abstracts from funded biomedical projects according to WHO-aligned research priorities. This task is crucial for monitoring research trends and identifying gaps in global health preparedness and response. Our approach builds on human-annotated projects, which are allocated one or more categories from a predefined list. A large language model is then used to generate `rationales' explaining the reasoning behind these annotations. This augmented data, comprising expert annotations and rationales, is subsequently used to fine-tune a smaller, more efficient model. Developed as part of the Pandemic PACT project, which aims to track and analyse research funding and clinical evidence for a wide range of diseases with outbreak potential, PPACE supports informed decision-making by research funders, policymakers, and independent researchers. We introduce and release both the trained model and the instruction-based dataset used for its training. Our evaluation shows that PPACE significantly outperforms its baselines. The release of PPACE and its associated dataset offers valuable resources for researchers in multilabel biomedical document classification and supports advancements in aligning biomedical research with key global health priorities.
△ Less
Submitted 19 July, 2024; v1 submitted 14 July, 2024;
originally announced July 2024.
-
DECISIVE Benchmarking Data Report: sUAS Performance Results from Phase I
Authors:
Adam Norton,
Reza Ahmadzadeh,
Kshitij Jerath,
Paul Robinette,
Jay Weitzen,
Thanuka Wickramarathne,
Holly Yanco,
Minseop Choi,
Ryan Donald,
Brendan Donoghue,
Christian Dumas,
Peter Gavriel,
Alden Giedraitis,
Brendan Hertel,
Jack Houle,
Nathan Letteri,
Edwin Meriaux,
Zahra Rezaei Khavas,
Rakshith Singh,
Gregg Willcox,
Naye Yoni
Abstract:
This report reviews all results derived from performance benchmarking conducted during Phase I of the Development and Execution of Comprehensive and Integrated Subterranean Intelligent Vehicle Evaluations (DECISIVE) project by the University of Massachusetts Lowell, using the test methods specified in the DECISIVE Test Methods Handbook v1.1 for evaluating small unmanned aerial systems (sUAS) perfo…
▽ More
This report reviews all results derived from performance benchmarking conducted during Phase I of the Development and Execution of Comprehensive and Integrated Subterranean Intelligent Vehicle Evaluations (DECISIVE) project by the University of Massachusetts Lowell, using the test methods specified in the DECISIVE Test Methods Handbook v1.1 for evaluating small unmanned aerial systems (sUAS) performance in subterranean and constrained indoor environments, spanning communications, field readiness, interface, obstacle avoidance, navigation, mapping, autonomy, trust, and situation awareness. Using those 20 test methods, over 230 tests were conducted across 8 sUAS platforms: Cleo Robotics Dronut X1P (P = prototype), FLIR Black Hornet PRS, Flyability Elios 2 GOV, Lumenier Nighthawk V3, Parrot ANAFI USA GOV, Skydio X2D, Teal Golden Eagle, and Vantage Robotics Vesper. Best in class criteria is specified for each applicable test method and the sUAS that match this criteria are named for each test method, including a high-level executive summary of their performance.
△ Less
Submitted 20 January, 2023; v1 submitted 18 January, 2023;
originally announced January 2023.
-
Contextual Autonomy Evaluation of Unmanned Aerial Vehicles in Subterranean Environments
Authors:
Ryan Donald,
Peter Gavriel,
Adam Norton,
S. Reza Ahmadzadeh
Abstract:
In this paper we focus on the evaluation of contextual autonomy for robots. More specifically, we propose a fuzzy framework for calculating the autonomy score for a small Unmanned Aerial Systems (sUAS) for performing a task while considering task complexity and environmental factors. Our framework is a cascaded Fuzzy Inference System (cFIS) composed of combination of three FIS which represent diff…
▽ More
In this paper we focus on the evaluation of contextual autonomy for robots. More specifically, we propose a fuzzy framework for calculating the autonomy score for a small Unmanned Aerial Systems (sUAS) for performing a task while considering task complexity and environmental factors. Our framework is a cascaded Fuzzy Inference System (cFIS) composed of combination of three FIS which represent different contextual autonomy capabilities. We performed several experiments to test our framework in various contexts, such as endurance time, navigation, take off/land, and room clearing, with seven different sUAS. We introduce a predictive measure which improves upon previous predictive measures, allowing for previous real-world task performance to be used in predicting future mission performance.
△ Less
Submitted 6 January, 2023;
originally announced January 2023.
-
DECISIVE Test Methods Handbook: Test Methods for Evaluating sUAS in Subterranean and Constrained Indoor Environments, Version 1.1
Authors:
Adam Norton,
Reza Ahmadzadeh,
Kshitij Jerath,
Paul Robinette,
Jay Weitzen,
Thanuka Wickramarathne,
Holly Yanco,
Minseop Choi,
Ryan Donald,
Brendan Donoghue,
Christian Dumas,
Peter Gavriel,
Alden Giedraitis,
Brendan Hertel,
Jack Houle,
Nathan Letteri,
Edwin Meriaux,
Zahra Rezaei Khavas,
Rakshith Singh,
Gregg Willcox,
Naye Yoni
Abstract:
This handbook outlines all test methods developed under the Development and Execution of Comprehensive and Integrated Subterranean Intelligent Vehicle Evaluations (DECISIVE) project by the University of Massachusetts Lowell for evaluating small unmanned aerial systems (sUAS) performance in subterranean and constrained indoor environments, spanning communications, field readiness, interface, obstac…
▽ More
This handbook outlines all test methods developed under the Development and Execution of Comprehensive and Integrated Subterranean Intelligent Vehicle Evaluations (DECISIVE) project by the University of Massachusetts Lowell for evaluating small unmanned aerial systems (sUAS) performance in subterranean and constrained indoor environments, spanning communications, field readiness, interface, obstacle avoidance, navigation, mapping, autonomy, trust, and situation awareness. For sUAS deployment in subterranean and constrained indoor environments, this puts forth two assumptions about applicable sUAS to be evaluated using these test methods: (1) able to operate without access to GPS signal, and (2) width from prop top to prop tip does not exceed 91 cm (36 in) wide (i.e., can physically fit through a typical doorway, although successful navigation through is not guaranteed). All test methods are specified using a common format: Purpose, Summary of Test Method, Apparatus and Artifacts, Equipment, Metrics, Procedure, and Example Data. All test methods are designed to be run in real-world environments (e.g., MOUT sites) or using fabricated apparatuses (e.g., test bays built from wood, or contained inside of one or more shipping containers).
△ Less
Submitted 20 January, 2023; v1 submitted 1 November, 2022;
originally announced November 2022.
-
Household Cloth Object Set: Fostering Benchmarking in Deformable Object Manipulation
Authors:
Irene Garcia-Camacho,
Júlia Borràs,
Berk Calli,
Adam Norton,
Guillem Alenyà
Abstract:
Benchmarking of robotic manipulations is one of the open issues in robotic research. An important factor that has enabled progress in this area in the last decade is the existence of common object sets that have been shared among different research groups. However, the existing object sets are very limited when it comes to cloth-like objects that have unique particularities and challenges. This pa…
▽ More
Benchmarking of robotic manipulations is one of the open issues in robotic research. An important factor that has enabled progress in this area in the last decade is the existence of common object sets that have been shared among different research groups. However, the existing object sets are very limited when it comes to cloth-like objects that have unique particularities and challenges. This paper is a first step towards the design of a cloth object set to be distributed among research groups from the robotics cloth manipulation community. We present a set of household cloth objects and related tasks that serve to expose the challenges related to gathering such an object set and propose a roadmap to the design of common benchmarks in cloth manipulation tasks, with the intention to set the grounds for a future debate in the community that will be necessary to foster benchmarking for the manipulation of cloth-like objects. Some RGB-D and object scans are also collected as examples for the objects in relevant configurations. More details about the cloth set are shared in http://www.iri.upc.edu/groups/perception/ClothObjectSet/HouseholdClothSet.html.
△ Less
Submitted 2 November, 2021;
originally announced November 2021.
-
Investigation of Multiple Resource Theory Design Principles on Robot Teleoperation and Workload Management
Authors:
Zhao Han,
Adam Norton,
Eric McCann,
Lisa Baraniecki,
Will Ober,
Dave Shane,
Anna Skinner,
Holly A. Yanco
Abstract:
Robot interfaces often only use the visual channel. Inspired by Wickens' Multiple Resource Theory, we investigated if the addition of audio elements would reduce cognitive workload and improve performance. Specifically, we designed a search and threat-defusal task (primary) with a memory test task (secondary). Eleven participants - predominantly first responders - were recruited to control a robot…
▽ More
Robot interfaces often only use the visual channel. Inspired by Wickens' Multiple Resource Theory, we investigated if the addition of audio elements would reduce cognitive workload and improve performance. Specifically, we designed a search and threat-defusal task (primary) with a memory test task (secondary). Eleven participants - predominantly first responders - were recruited to control a robot to clear all threats in a combination of four conditions of primary and secondary tasks in visual and auditory channels. We did not find any statistically significant differences in performance or workload across subjects, making it questionable that Multiple Resource Theory could shorten longer-term task completion time and reduce workload. Our results suggest that considering individual differences for splitting interface modalities across multiple channels requires further investigation.
△ Less
Submitted 30 March, 2021; v1 submitted 30 March, 2021;
originally announced March 2021.
-
Benchmarking Off-The-Shelf Solutions to Robotic Assembly Tasks
Authors:
Wenzhao Lian,
Tim Kelch,
Dirk Holz,
Adam Norton,
Stefan Schaal
Abstract:
In recent years, many learning based approaches have been studied to realize robotic manipulation and assembly tasks, often including vision and force/tactile feedback. However, it remains frequently unclear what is the baseline state-of-the-art performance and what are the bottleneck problems. In this work, we evaluate some off-the-shelf (OTS) industrial solutions on a recently introduced benchma…
▽ More
In recent years, many learning based approaches have been studied to realize robotic manipulation and assembly tasks, often including vision and force/tactile feedback. However, it remains frequently unclear what is the baseline state-of-the-art performance and what are the bottleneck problems. In this work, we evaluate some off-the-shelf (OTS) industrial solutions on a recently introduced benchmark, the National Institute of Standards and Technology (NIST) Assembly Task Boards. A set of assembly tasks are introduced and baseline methods are provided to understand their intrinsic difficulty. Multiple sensor-based robotic solutions are then evaluated, including hybrid force/motion control and 2D/3D pattern matching algorithms. An end-to-end integrated solution that accomplishes the tasks is also provided. The results and findings throughout the study reveal a few noticeable factors that impede the adoptions of the OTS solutions: expertise dependent, limited applicability, lack of interoperability, no scene awareness or error recovery mechanisms, and high cost. This paper also provides a first attempt of an objective benchmark performance on the NIST Assembly Task Boards as a reference comparison for future works on this problem.
△ Less
Submitted 8 March, 2021;
originally announced March 2021.
-
Design, Modelling and Validation of a Novel Extra Slender Continuum Robot for In-situ Inspection and Repair in Aeroengine
Authors:
Mingfeng Wang,
Xin Dong,
Weiming Ba,
Abdelkhalick Mohammad,
Dragos Axinte,
Andy Norton
Abstract:
In-situ aeroengine maintenance works are highly beneficial as it can significantly reduce the current maintenance cycle which is extensive and costly due to the disassembly requirement of engines from aircrafts. However, navigating in/out via inspection ports and performing multi-axis movements with end-effectors in constrained environments (e.g. combustion chamber) are fairly challenging. A novel…
▽ More
In-situ aeroengine maintenance works are highly beneficial as it can significantly reduce the current maintenance cycle which is extensive and costly due to the disassembly requirement of engines from aircrafts. However, navigating in/out via inspection ports and performing multi-axis movements with end-effectors in constrained environments (e.g. combustion chamber) are fairly challenging. A novel extra-slender (diameter-to-length ratio <0.02) dual-stage continuum robot (16 degree-of-freedom) is proposed to navigate in/out confined environments and perform required configuration shapes for further repair operations. Firstly, the robot design presents several innovative mechatronic solutions: (i) dual-stage tendon-driven structure with bevelled disks to perform required shapes and to provide selective stiffness for carrying high payloads; (ii) various rigid-compliant combined joints to enable different flexibility and stiffness in each stage; (iii) three commanding cables for each 2-DoF section to minimise the number of actuators with precise actuations. Secondly, a segment-scaled piecewise-constant-curvature-theory based kinematic model and a Kirchhoff-elastic-rod-theory based static model are established by considering the applied forces/moments (friction, actuation, gravity and external load), where the friction coefficient is modelled as a function of bending angle. Finally, experiments were carried out to validate the proposed static modelling and to evaluate the robot capabilities of performing the predefined shape and stiffness.
△ Less
Submitted 9 October, 2019;
originally announced October 2019.
-
Towards A Robot Explanation System: A Survey and Our Approach to State Summarization, Storage and Querying, and Human Interface
Authors:
Zhao Han,
Jordan Allspaw,
Adam Norton,
Holly A. Yanco
Abstract:
As robot systems become more ubiquitous, developing understandable robot systems becomes increasingly important in order to build trust. In this paper, we present an approach to developing a holistic robot explanation system, which consists of three interconnected components: state summarization, storage and querying, and human interface. To find trends towards and gaps in the development of such…
▽ More
As robot systems become more ubiquitous, developing understandable robot systems becomes increasingly important in order to build trust. In this paper, we present an approach to developing a holistic robot explanation system, which consists of three interconnected components: state summarization, storage and querying, and human interface. To find trends towards and gaps in the development of such an integrated system, a literature review was performed and categorized around those three components, with a focus on robotics applications. After the review of each component, we discuss our proposed approach for robot explanation. Finally, we summarize the system as a whole and review its functionality.
△ Less
Submitted 13 September, 2019;
originally announced September 2019.
-
Adversarial-Playground: A Visualization Suite Showing How Adversarial Examples Fool Deep Learning
Authors:
Andrew P. Norton,
Yanjun Qi
Abstract:
Recent studies have shown that attackers can force deep learning models to misclassify so-called "adversarial examples": maliciously generated images formed by making imperceptible modifications to pixel values. With growing interest in deep learning for security applications, it is important for security experts and users of machine learning to recognize how learning systems may be attacked. Due…
▽ More
Recent studies have shown that attackers can force deep learning models to misclassify so-called "adversarial examples": maliciously generated images formed by making imperceptible modifications to pixel values. With growing interest in deep learning for security applications, it is important for security experts and users of machine learning to recognize how learning systems may be attacked. Due to the complex nature of deep learning, it is challenging to understand how deep models can be fooled by adversarial examples. Thus, we present a web-based visualization tool, Adversarial-Playground, to demonstrate the efficacy of common adversarial methods against a convolutional neural network (CNN) system. Adversarial-Playground is educational, modular and interactive. (1) It enables non-experts to compare examples visually and to understand why an adversarial example can fool a CNN-based image classifier. (2) It can help security experts explore more vulnerability of deep learning as a software module. (3) Building an interactive visualization is challenging in this domain due to the large feature space of image classification (generating adversarial examples is slow in general and visualizing images are costly). Through multiple novel design choices, our tool can provide fast and accurate responses to user requests. Empirically, we find that our client-server division strategy reduced the response time by an average of 1.5 seconds per sample. Our other innovation, a faster variant of JSMA evasion algorithm, empirically performed twice as fast as JSMA and yet maintains a comparable evasion rate.
Project source code and data from our experiments available at: https://github.com/QData/AdversarialDNN-Playground
△ Less
Submitted 1 August, 2017;
originally announced August 2017.
-
Adversarial-Playground: A Visualization Suite for Adversarial Sample Generation
Authors:
Andrew Norton,
Yanjun Qi
Abstract:
With growing interest in adversarial machine learning, it is important for machine learning practitioners and users to understand how their models may be attacked. We propose a web-based visualization tool, Adversarial-Playground, to demonstrate the efficacy of common adversarial methods against a deep neural network (DNN) model, built on top of the TensorFlow library. Adversarial-Playground provi…
▽ More
With growing interest in adversarial machine learning, it is important for machine learning practitioners and users to understand how their models may be attacked. We propose a web-based visualization tool, Adversarial-Playground, to demonstrate the efficacy of common adversarial methods against a deep neural network (DNN) model, built on top of the TensorFlow library. Adversarial-Playground provides users an efficient and effective experience in exploring techniques generating adversarial examples, which are inputs crafted by an adversary to fool a machine learning system. To enable Adversarial-Playground to generate quick and accurate responses for users, we use two primary tactics: (1) We propose a faster variant of the state-of-the-art Jacobian saliency map approach that maintains a comparable evasion rate. (2) Our visualization does not transmit the generated adversarial images to the client, but rather only the matrix describing the sample and the vector representing classification likelihoods.
The source code along with the data from all of our experiments are available at \url{https://github.com/QData/AdversarialDNN-Playground}.
△ Less
Submitted 16 June, 2017; v1 submitted 6 June, 2017;
originally announced June 2017.
