Showing 1–50 of 51 results for author: Haddadin, S

Optimizing Interaction Space: Enlarging the Capture Volume for Multiple Portable Motion Capture Devices

Authors: Muhammad Hilman Fatoni, Christopher Herneth, Junnan Li, Fajar Budiman, Amartya Ganguly, Sami Haddadin

Abstract: Markerless motion capture devices such as the Leap Motion Controller (LMC) have been extensively used for tracking hand, wrist, and forearm positions as an alternative to Marker-based Motion Capture (MMC). However, previous studies have highlighted the subpar performance of LMC in reliably recording hand kinematics. In this study, we employ four LMC devices to optimize their collective tracking vo… ▽ More Markerless motion capture devices such as the Leap Motion Controller (LMC) have been extensively used for tracking hand, wrist, and forearm positions as an alternative to Marker-based Motion Capture (MMC). However, previous studies have highlighted the subpar performance of LMC in reliably recording hand kinematics. In this study, we employ four LMC devices to optimize their collective tracking volume, aiming to enhance the accuracy and precision of hand kinematics. Through Monte Carlo simulation, we determine an optimized layout for the four LMC devices and subsequently conduct reliability and validity experiments encompassing 1560 trials across ten subjects. The combined tracking volume is validated against an MMC system, particularly for kinematic movements involving wrist, index, and thumb flexion. Utilizing calculation resources in one computer, our result of the optimized configuration has a better visibility rate with a value of 0.05 $\pm$ 0.55 compared to the initial configuration with -0.07 $\pm$ 0.40. Multiple Leap Motion Controllers (LMCs) have proven to increase the interaction space of capture volume but are still unable to give agreeable measurements from dynamic movement. △ Less

Submitted 30 August, 2024; originally announced August 2024.

Comments: This paper has eight pages and five figures. It has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible. The code used in this work is available at https://github.com/hilmanfatoni/Multi-LMC_Optimization

Time-Optimized Trajectory Planning for Non-Prehensile Object Transportation in 3D

Authors: Lingyun Chen, Haoyu Yu, Abdeldjallil Naceri, Abdalla Swikir, Sami Haddadin

Abstract: Non-prehensile object transportation offers a way to enhance robotic performance in object manipulation tasks, especially with unstable objects. Effective trajectory planning requires simultaneous consideration of robot motion constraints and object stability. Here, we introduce a physical model for object stability and propose a novel trajectory planning approach for non-prehensile transportation… ▽ More Non-prehensile object transportation offers a way to enhance robotic performance in object manipulation tasks, especially with unstable objects. Effective trajectory planning requires simultaneous consideration of robot motion constraints and object stability. Here, we introduce a physical model for object stability and propose a novel trajectory planning approach for non-prehensile transportation along arbitrary straight lines in 3D space. Validation with a 7-DoF Franka Panda robot confirms improved transportation speed via tray rotation integration while ensuring object stability and robot motion constraints. △ Less

Submitted 29 August, 2024; originally announced August 2024.

Comments: Accepted to the European Robotic Forum (ERF) 2024

Functional kinematic and kinetic requirements of the upper limb during activities of daily living: a recommendation on necessary joint capabilities for prosthetic arms

Authors: Christopher Herneth, Amartya Ganguly, Sami Haddadin

Abstract: Prosthetic limb abandonment remains an unsolved challenge as amputees consistently reject their devices. Current prosthetic designs often fail to balance human-like perfomance with acceptable device weight, highlighting the need for optimised designs tailored to modern tasks. This study aims to provide a comprehensive dataset of joint kinematics and kinetics essential for performing activities of… ▽ More Prosthetic limb abandonment remains an unsolved challenge as amputees consistently reject their devices. Current prosthetic designs often fail to balance human-like perfomance with acceptable device weight, highlighting the need for optimised designs tailored to modern tasks. This study aims to provide a comprehensive dataset of joint kinematics and kinetics essential for performing activities of daily living (ADL), thereby informing the design of more functional and user-friendly prosthetic devices. Functionally required Ranges of Motion (ROM), velocities, and torques for the Glenohumeral (rotation), elbow, Radioulnar, and wrist joints were computed using motion capture data from 12 subjects performing 24 ADLs. Our approach included the computation of joint torques for varying mass and inertia properties of the upper limb, while torques induced by the manipulation of experimental objects were considered by their interaction wrench with the subjects hand. Joint torques pertaining to individual ADL scaled linearly with limb and object mass and mass distribution, permitting their generalisation to not explicitly simulated limb and object dynamics with linear regressors (LRM), exhibiting coefficients of determination R = 0.99 pm 0.01. Exemplifying an application of data-driven prosthesis design, we optimise wrist axes orientations for two serial and two differential joint configurations. Optimised axes reduced peak power requirements, between 22 to 38 percent compared to anatomical configurations, by exploiting high torque correlations between Ulnar deviation and wrist flexion/extension joints. This study offers critical insights into the functional requirements of upper limb prostheses, providing a valuable foundation for data-driven prosthetic design that addresses key user concerns and enhances device adoption. △ Less

Submitted 26 August, 2024; originally announced August 2024.

Comments: Accepted at IROS 2024

ACM Class: J.2

Visuo-Tactile Exploration of Unknown Rigid 3D Curvatures by Vision-Augmented Unified Force-Impedance Control

Authors: Kübra Karacan, Anran Zhang, Hamid Sadeghian, Fan Wu, Sami Haddadin

Abstract: Despite recent advancements in torque-controlled tactile robots, integrating them into manufacturing settings remains challenging, particularly in complex environments. Simplifying robotic skill programming for non-experts is crucial for increasing robot deployment in manufacturing. This work proposes an innovative approach, Vision-Augmented Unified Force-Impedance Control (VA-UFIC), aimed at intu… ▽ More Despite recent advancements in torque-controlled tactile robots, integrating them into manufacturing settings remains challenging, particularly in complex environments. Simplifying robotic skill programming for non-experts is crucial for increasing robot deployment in manufacturing. This work proposes an innovative approach, Vision-Augmented Unified Force-Impedance Control (VA-UFIC), aimed at intuitive visuo-tactile exploration of unknown 3D curvatures. VA-UFIC stands out by seamlessly integrating vision and tactile data, enabling the exploration of diverse contact shapes in three dimensions, including point contacts, flat contacts with concave and convex curvatures, and scenarios involving contact loss. A pivotal component of our method is a robust online contact alignment monitoring system that considers tactile error, local surface curvature, and orientation, facilitating adaptive adjustments of robot stiffness and force regulation during exploration. We introduce virtual energy tanks within the control framework to ensure safety and stability, effectively addressing inherent safety concerns in visuo-tactile exploration. Evaluation using a Franka Emika research robot demonstrates the efficacy of VA-UFIC in exploring unknown 3D curvatures while adhering to arbitrarily defined force-motion policies. By seamlessly integrating vision and tactile sensing, VA-UFIC offers a promising avenue for intuitive exploration of complex environments, with potential applications spanning manufacturing, inspection, and beyond. △ Less

Submitted 26 August, 2024; originally announced August 2024.

Comments: 8 pages, 3 figures, accepted by IROS 2024

Identification and validation of the dynamic model of a tendon-driven anthropomorphic finger

Authors: Junnan Li, Lingyun Chen, Johannes Ringwald, Edmundo Pozo Fortunic, Amartya Ganguly, Sami Haddadin

Abstract: This study addresses the absence of an identification framework to quantify a comprehensive dynamic model of human and anthropomorphic tendon-driven fingers, which is necessary to investigate the physiological properties of human fingers and improve the control of robotic hands. First, a generalized dynamic model was formulated, which takes into account the inherent properties of such a mechanical… ▽ More This study addresses the absence of an identification framework to quantify a comprehensive dynamic model of human and anthropomorphic tendon-driven fingers, which is necessary to investigate the physiological properties of human fingers and improve the control of robotic hands. First, a generalized dynamic model was formulated, which takes into account the inherent properties of such a mechanical system. This includes rigid-body dynamics, coupling matrix, joint viscoelasticity, and tendon friction. Then, we propose a methodology comprising a series of experiments, for step-wise identification and validation of this dynamic model. Moreover, an experimental setup was designed and constructed that features actuation modules and peripheral sensors to facilitate the identification process. To verify the proposed methodology, a 3D-printed robotic finger based on the index finger design of the Dexmart hand was developed, and the proposed experiments were executed to identify and validate its dynamic model. This study could be extended to explore the identification of cadaver hands, aiming for a consistent dataset from a single cadaver specimen to improve the development of musculoskeletal hand models. △ Less

Submitted 23 August, 2024; originally announced August 2024.

Comments: 8 pages, 9 figures

Tactile-Morph Skills: Energy-Based Control Meets Data-Driven Learning

Authors: Anran Zhang, Kübra Karacan, Hamid Sadeghian, Yansong Wu, Fan Wu, Sami Haddadin

Abstract: Robotic manipulation is essential for modernizing factories and automating industrial tasks like polishing, which require advanced tactile abilities. These robots must be easily set up, safely work with humans, learn tasks autonomously, and transfer skills to similar tasks. Addressing these needs, we introduce the tactile-morph skill framework, which integrates unified force-impedance control with… ▽ More Robotic manipulation is essential for modernizing factories and automating industrial tasks like polishing, which require advanced tactile abilities. These robots must be easily set up, safely work with humans, learn tasks autonomously, and transfer skills to similar tasks. Addressing these needs, we introduce the tactile-morph skill framework, which integrates unified force-impedance control with data-driven learning. Our system adjusts robot movements and force application based on estimated energy levels for the desired trajectory and force profile, ensuring safety by stopping if energy allocated for the control runs out. Using a Temporal Convolutional Network, we estimate the energy distribution for a given motion and force profile, enabling skill transfer across different tasks and surfaces. Our approach maintains stability and performance even on unfamiliar geometries with similar friction characteristics, demonstrating improved accuracy, zero-shot transferable performance, and enhanced safety in real-world scenarios. This framework promises to enhance robotic capabilities in industrial settings, making intelligent robots more accessible and valuable. △ Less

Submitted 23 August, 2024; v1 submitted 22 August, 2024; originally announced August 2024.

Comments: 15 pages, 7 figures,updated footnote

Object Augmentation Algorithm: Computing virtual object motion and object induced interaction wrench from optical markers

Authors: Christopher Herneth, Junnan Li, Muhammad Hilman Fatoni, Amartya Ganguly, Sami Haddadin

Abstract: This study addresses the critical need for diverse and comprehensive data focused on human arm joint torques while performing activities of daily living (ADL). Previous studies have often overlooked the influence of objects on joint torques during ADL, resulting in limited datasets for analysis. To address this gap, we propose an Object Augmentation Algorithm (OAA) capable of augmenting existing m… ▽ More This study addresses the critical need for diverse and comprehensive data focused on human arm joint torques while performing activities of daily living (ADL). Previous studies have often overlooked the influence of objects on joint torques during ADL, resulting in limited datasets for analysis. To address this gap, we propose an Object Augmentation Algorithm (OAA) capable of augmenting existing marker-based databases with virtual object motions and object-induced joint torque estimations. The OAA consists of five phases: (1) computing hand coordinate systems from optical markers, (2) characterising object movements with virtual markers, (3) calculating object motions through inverse kinematics (IK), (4) determining the wrench necessary for prescribed object motion using inverse dynamics (ID), and (5) computing joint torques resulting from object manipulation. The algorithm's accuracy is validated through trajectory tracking and torque analysis on a 7+4 degree of freedom (DoF) robotic hand-arm system, manipulating three unique objects. The results show that the OAA can accurately and precisely estimate 6 DoF object motion and object-induced joint torques. Correlations between computed and measured quantities were > 0.99 for object trajectories and > 0.93 for joint torques. The OAA was further shown to be robust to variations in the number and placement of input markers, which are expected between databases. Differences between repeated experiments were minor but significant (p < 0.05). The algorithm expands the scope of available data and facilitates more comprehensive analyses of human-object interaction dynamics. △ Less

Submitted 27 August, 2024; v1 submitted 14 August, 2024; originally announced August 2024.

Comments: An open source implementation of the described algorithm is available at https://github.com/ChristopherHerneth/ObjectAugmentationAlgorithm/tree/main. Accompanying video material may be found here https://youtu.be/8oz-awvyNRA. The article was accepted at IROS 2024

MSC Class: J.3

Towards Unconstrained Collision Injury Protection Data Sets: Initial Surrogate Experiments for the Human Hand

Authors: Robin Jeanne Kirschner, Jinyu Yang, Edonis Elshani, Carina M. Micheler, Tobias Leibbrand, Dirk Müller, Claudio Glowalla, Nader Rajaei, Rainer Burgkart, Sami Haddadin

Abstract: Safety for physical human-robot interaction (pHRI) is a major concern for all application domains. While current standardization for industrial robot applications provide safety constraints that address the onset of pain in blunt impacts, these impact thresholds are difficult to use on edged or pointed impactors. The most severe injuries occur in constrained contact scenarios, where crushing is po… ▽ More Safety for physical human-robot interaction (pHRI) is a major concern for all application domains. While current standardization for industrial robot applications provide safety constraints that address the onset of pain in blunt impacts, these impact thresholds are difficult to use on edged or pointed impactors. The most severe injuries occur in constrained contact scenarios, where crushing is possible. Nevertheless, situations potentially resulting in constrained contact only occur in certain areas of a workspace and design or organisational approaches can be used to avoid them. What remains are risks to the human physical integrity caused by unconstrained accidental contacts, which are difficult to avoid while maintaining robot motion efficiency. Nevertheless, the probability and severity of injuries occurring with edged or pointed impacting objects in unconstrained collisions is hardly researched. In this paper, we propose an experimental setup and procedure using two pendulums modeling human hands and arms and robots to understand the injury potential of unconstrained collisions of human hands with edged objects. Pig feet are used as ex vivo surrogate samples - as these closely resemble the physiological characteristics of human hands - to create an initial injury database on the severity of injuries caused by unconstrained edged or pointed impacts. For the effective mass range of typical lightweight robots, the data obtained show low probabilities of injuries such as skin cuts or bone/tendon injuries in unconstrained collisions when the velocity is reduced to < 0.5 m/s. The proposed experimental setups and procedures should be complemented by sufficient human modeling and will eventually lead to a complete understanding of the biomechanical injury potential in pHRI. △ Less

Submitted 18 August, 2024; v1 submitted 12 August, 2024; originally announced August 2024.

Comments: \c{opyright} 2024 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works

OPENGRASP-LITE Version 1.0: A Tactile Artificial Hand with a Compliant Linkage Mechanism

Authors: Sonja Groß, Michael Ratzel, Edgar Welte, Diego Hidalgo-Carvajal, Lingyun Chen, Edmundo Pozo Fortunić, Amartya Ganguly, Abdalla Swikir, Sami Haddadin

Abstract: Recent research has seen notable progress in the development of linkage-based artificial hands. While previous designs have focused on adaptive grasping, dexterity and biomimetic artificial skin, only a few systems have proposed a lightweight, accessible solution integrating tactile sensing with a compliant linkage-based mechanism. This paper introduces OPENGRASP LITE, an open-source, highly integ… ▽ More Recent research has seen notable progress in the development of linkage-based artificial hands. While previous designs have focused on adaptive grasping, dexterity and biomimetic artificial skin, only a few systems have proposed a lightweight, accessible solution integrating tactile sensing with a compliant linkage-based mechanism. This paper introduces OPENGRASP LITE, an open-source, highly integrated, tactile, and lightweight artificial hand. Leveraging compliant linkage systems and MEMS barometer-based tactile sensing, it offers versatile grasping capabilities with six degrees of actuation. By providing tactile sensors and enabling soft grasping, it serves as an accessible platform for further research in tactile artificial hands. △ Less

Submitted 5 August, 2024; originally announced August 2024.

Comments: Accepted at IEEE/RSJ International Conference on Intelligent Robots and Systems, 14-18 October 2024

Autonomous and Teleoperation Control of a Drawing Robot Avatar

Authors: Lingyun Chen, Abdeldjallil Naceri, Abdalla Swikir, Sandra Hirche, Sami Haddadin

Abstract: A drawing robot avatar is a robotic system that allows for telepresence-based drawing, enabling users to remotely control a robotic arm and create drawings in real-time from a remote location. The proposed control framework aims to improve bimanual robot telepresence quality by reducing the user workload and required prior knowledge through the automation of secondary or auxiliary tasks. The intro… ▽ More A drawing robot avatar is a robotic system that allows for telepresence-based drawing, enabling users to remotely control a robotic arm and create drawings in real-time from a remote location. The proposed control framework aims to improve bimanual robot telepresence quality by reducing the user workload and required prior knowledge through the automation of secondary or auxiliary tasks. The introduced novel method calculates the near-optimal Cartesian end-effector pose in terms of visual feedback quality for the attached eye-to-hand camera with motion constraints in consideration. The effectiveness is demonstrated by conducting user studies of drawing reference shapes using the implemented robot avatar compared to stationary and teleoperated camera pose conditions. Our results demonstrate that the proposed control framework offers improved visual feedback quality and drawing performance. △ Less

Submitted 29 July, 2024; originally announced July 2024.

Comments: Accepted to ICRA 2024

Optimal Control for Clutched-Elastic Robots: A Contact-Implicit Approach

Authors: Dennis Ossadnik, Vasilije Rakčević, Mehmet C. Yildirim, Edmundo Pozo Fortunić, Hugo T. M. Kussaba, Abdalla Swikir, Sami Haddadin

Abstract: Intrinsically elastic robots surpass their rigid counterparts in a range of different characteristics. By temporarily storing potential energy and subsequently converting it to kinetic energy, elastic robots are capable of highly dynamic motions even with limited motor power. However, the time-dependency of this energy storage and release mechanism remains one of the major challenges in controllin… ▽ More Intrinsically elastic robots surpass their rigid counterparts in a range of different characteristics. By temporarily storing potential energy and subsequently converting it to kinetic energy, elastic robots are capable of highly dynamic motions even with limited motor power. However, the time-dependency of this energy storage and release mechanism remains one of the major challenges in controlling elastic robots. A possible remedy is the introduction of locking elements (i.e. clutches and brakes) in the drive train. This gives rise to a new class of robots, so-called clutched-elastic robots (CER), with which it is possible to precisely control the energy-transfer timing. A prevalent challenge in the realm of CERs is the automatic discovery of clutch sequences. Due to complexity, many methods still rely on pre-defined modes. In this paper, we introduce a novel contact-implicit scheme designed to optimize both control input and clutch sequence simultaneously. A penalty in the objective function ensures the prevention of unnecessary clutch transitions. We empirically demonstrate the effectiveness of our proposed method on a double pendulum equipped with two of our newly proposed clutch-based Bi-Stiffness Actuators (BSA). △ Less

Submitted 29 July, 2024; v1 submitted 17 July, 2024; originally announced July 2024.

Comments: Accepted to the 2024 IEEE International Conference on Robotics and Automation (ICRA 2024). The first two authors contributed equally to this work. A cited paper, "Posa M, Cantu C, and Tedrake R (2014)" has a corrigendum (https://doi.org/10.1177/0278364914533878) referencing earlier work by Dr. Kerim Yunt

MSC Class: 49N90 ACM Class: I.2.9

SVan: A Mobile Hub as a Field Robotics Development and Deployment Platform

Authors: Alexander Moortgat-Pick, Anna Adamczyk, Daniel A Duecker, Sami Haddadin

Abstract: As robotics becomes increasingly vital for environmental protection, there is a growing need for effective deployment methods that match the pace of robotics innovation. Current strategies often fall short, leaving a gap between the potential of robotics and their practical application in the field. Addressing this challenge, we introduce a mobile hub concept designed to provide the necessary infr… ▽ More As robotics becomes increasingly vital for environmental protection, there is a growing need for effective deployment methods that match the pace of robotics innovation. Current strategies often fall short, leaving a gap between the potential of robotics and their practical application in the field. Addressing this challenge, we introduce a mobile hub concept designed to provide the necessary infrastructure and support for deploying a diverse, multi-domain robot team effectively. This paper presents the development and insights into `SVAN' (Synchronous Team-Robot Van), a prototype of our mobile hub concept. We delve into the mechanical construction and software setup of SVAN, offering a comprehensive overview of its capabilities and design considerations. Further, we discuss the hardware specifications and share valuable lessons learned during the prototype's development and deployment. In addition to this paper, an accepted video complements our exploration by depicting SVAN in its envisioned role as an environmental guardian, highlighting its potential in ecological monitoring and preservation. Furthermore, our discussion is enriched by referencing a previously accepted paper detailing a novel methodology for continuous UAV mission cycling enabled by a mobile hub like SVAN. These accompanying works underscore our contribution towards addressing the existing gaps in robot deployment strategies, presenting a scalable and efficient framework to overcome operational challenges in environmental robotics. △ Less

Submitted 6 May, 2024; originally announced May 2024.

Comments: 7 pages, 10 figures, accepted paper at IEEE ICRA Workshop on Field Robotics 2024

Towards Safe Robot Use with Edged or Pointed Objects: A Surrogate Study Assembling a Human Hand Injury Protection Database

Authors: Robin Jeanne Kirschner, Carina M. Micheler, Yangcan Zhou, Sebastian Siegner, Mazin Hamad, Claudio Glowalla, Jan Neumann, Nader Rajaei, Rainer Burgkart, Sami Haddadin

Abstract: The use of pointed or edged tools or objects is one of the most challenging aspects of today's application of physical human-robot interaction (pHRI). One reason for this is that the severity of harm caused by such edged or pointed impactors is less well studied than for blunt impactors. Consequently, the standards specify well-reasoned force and pressure thresholds for blunt impactors and advise… ▽ More The use of pointed or edged tools or objects is one of the most challenging aspects of today's application of physical human-robot interaction (pHRI). One reason for this is that the severity of harm caused by such edged or pointed impactors is less well studied than for blunt impactors. Consequently, the standards specify well-reasoned force and pressure thresholds for blunt impactors and advise avoiding any edges and corners in contacts. Nevertheless, pointed or edged impactor geometries cannot be completely ruled out in real pHRI applications. For example, to allow edged or pointed tools such as screwdrivers near human operators, the knowledge of injury severity needs to be extended so that robot integrators can perform well-reasoned, time-efficient risk assessments. In this paper, we provide the initial datasets on injury prevention for the human hand based on drop tests with surrogates for the human hand, namely pig claws and chicken drumsticks. We then demonstrate the ease and efficiency of robot use using the dataset for contact on two examples. Finally, our experiments provide a set of injuries that may also be expected for human subjects under certain robot mass-velocity constellations in collisions. To extend this work, testing on human samples and a collaborative effort from research institutes worldwide is needed to create a comprehensive human injury avoidance database for any pHRI scenario and thus for safe pHRI applications including edged and pointed geometries. △ Less

Submitted 18 August, 2024; v1 submitted 5 April, 2024; originally announced April 2024.

Comments: \c{opyright} 2024 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works

Learning Barrier-Certified Polynomial Dynamical Systems for Obstacle Avoidance with Robots

Authors: Martin Schonger, Hugo T. M. Kussaba, Lingyun Chen, Luis Figueredo, Abdalla Swikir, Aude Billard, Sami Haddadin

Abstract: Established techniques that enable robots to learn from demonstrations are based on learning a stable dynamical system (DS). To increase the robots' resilience to perturbations during tasks that involve static obstacle avoidance, we propose incorporating barrier certificates into an optimization problem to learn a stable and barrier-certified DS. Such optimization problem can be very complex or ex… ▽ More Established techniques that enable robots to learn from demonstrations are based on learning a stable dynamical system (DS). To increase the robots' resilience to perturbations during tasks that involve static obstacle avoidance, we propose incorporating barrier certificates into an optimization problem to learn a stable and barrier-certified DS. Such optimization problem can be very complex or extremely conservative when the traditional linear parameter-varying formulation is used. Thus, different from previous approaches in the literature, we propose to use polynomial representations for DSs, which yields an optimization problem that can be tackled by sum-of-squares techniques. Finally, our approach can handle obstacle shapes that fall outside the scope of assumptions typically found in the literature concerning obstacle avoidance within the DS learning framework. Supplementary material can be found at the project webpage: https://martinschonger.github.io/abc-ds △ Less

Submitted 12 March, 2024; originally announced March 2024.

Comments: 7 pages, 7 figures, accepted to the 2024 IEEE International Conference on Robotics and Automation (ICRA 2024)

MSC Class: 68T40 ACM Class: I.2.9

Safe Execution of Learned Orientation Skills with Conic Control Barrier Functions

Authors: Zheng Shen, Matteo Saveriano, Fares J. Abu-Dakka, Sami Haddadin

Abstract: In the field of Learning from Demonstration (LfD), Dynamical Systems (DSs) have gained significant attention due to their ability to generate real-time motions and reach predefined targets. However, the conventional convergence-centric behavior exhibited by DSs may fall short in safety-critical tasks, specifically, those requiring precise replication of demonstrated trajectories or strict adherenc… ▽ More In the field of Learning from Demonstration (LfD), Dynamical Systems (DSs) have gained significant attention due to their ability to generate real-time motions and reach predefined targets. However, the conventional convergence-centric behavior exhibited by DSs may fall short in safety-critical tasks, specifically, those requiring precise replication of demonstrated trajectories or strict adherence to constrained regions even in the presence of perturbations or human intervention. Moreover, existing DS research often assumes demonstrations solely in Euclidean space, overlooking the crucial aspect of orientation in various applications. To alleviate these shortcomings, we present an innovative approach geared toward ensuring the safe execution of learned orientation skills within constrained regions surrounding a reference trajectory. This involves learning a stable DS on SO(3), extracting time-varying conic constraints from the variability observed in expert demonstrations, and bounding the evolution of the DS with Conic Control Barrier Function (CCBF) to fulfill the constraints. We validated our approach through extensive evaluation in simulation and showcased its effectiveness for a cutting skill in the context of assisted teleoperation. △ Less

Submitted 8 March, 2024; originally announced March 2024.

Geometric Slosh-Free Tracking for Robotic Manipulators

Authors: Jon Arrizabalaga, Lukas Pries, Riddhiman Laha, Runkang Li, Sami Haddadin, Markus Ryll

Abstract: This work focuses on the agile transportation of liquids with robotic manipulators. In contrast to existing methods that are either computationally heavy, system/container specific or dependant on a singularity-prone pendulum model, we present a real-time slosh-free tracking technique. This method solely requires the reference trajectory and the robot's kinematic constraints to output kinematicall… ▽ More This work focuses on the agile transportation of liquids with robotic manipulators. In contrast to existing methods that are either computationally heavy, system/container specific or dependant on a singularity-prone pendulum model, we present a real-time slosh-free tracking technique. This method solely requires the reference trajectory and the robot's kinematic constraints to output kinematically feasible joint space commands. The crucial element underlying this approach consists on mimicking the end-effector's motion through a virtual quadrotor, which is inherently slosh-free and differentially flat, thereby allowing us to calculate a slosh-free reference orientation. Through the utilization of a cascaded proportional-derivative (PD) controller, this slosh-free reference is transformed into task space acceleration commands, which, following the resolution of a Quadratic Program (QP) based on Resolved Acceleration Control (RAC), are translated into a feasible joint configuration. The validity of the proposed approach is demonstrated by simulated and real-world experiments on a 7 DoF Franka Emika Panda robot. Code: https://github.com/jonarriza96/gsft Video: https://youtu.be/4kitqYVS9n8 △ Less

Submitted 7 February, 2024; originally announced February 2024.

Comments: This paper has been accepted for publication at the IEEE International Conference on Robotics and Automation (ICRA), Yokohama, Japan, May 2024. Copyright @ IEEE

Smooth real-time motion planning based on a cascade dual-quaternion screw-geometry MPC

Authors: Ainoor Teimoorzadeh, Frederico Fernandes Afonso Silva, Luis F. C. Figueredo, Sami Haddadin

Abstract: This paper investigates the tracking problem of a smooth coordinate-invariant trajectory using dual quaternion algebra. The proposed architecture consists of a cascade structure in which the outer-loop MPC performs real-time smoothing of the manipulator's end-effector twist while an inner-loop kinematic controller ensures tracking of the instantaneous desired end-effector pose. Experiments on a… ▽ More This paper investigates the tracking problem of a smooth coordinate-invariant trajectory using dual quaternion algebra. The proposed architecture consists of a cascade structure in which the outer-loop MPC performs real-time smoothing of the manipulator's end-effector twist while an inner-loop kinematic controller ensures tracking of the instantaneous desired end-effector pose. Experiments on a $7$-DoF Franka Emika Panda robotic manipulator validate the proposed method demonstrating its application to constraint the robot twists, accelerations and jerks within prescribed bounds. △ Less

Submitted 7 February, 2024; originally announced February 2024.

Real-time Contact State Estimation in Shape Control of Deformable Linear Objects under Small Environmental Constraints

Authors: Kejia Chen, Zhenshan Bing, Yansong Wu, Fan Wu, Liding Zhang, Sami Haddadin, Alois Knoll

Abstract: Controlling the shape of deformable linear objects using robots and constraints provided by environmental fixtures has diverse industrial applications. In order to establish robust contacts with these fixtures, accurate estimation of the contact state is essential for preventing and rectifying potential anomalies. However, this task is challenging due to the small sizes of fixtures, the requiremen… ▽ More Controlling the shape of deformable linear objects using robots and constraints provided by environmental fixtures has diverse industrial applications. In order to establish robust contacts with these fixtures, accurate estimation of the contact state is essential for preventing and rectifying potential anomalies. However, this task is challenging due to the small sizes of fixtures, the requirement for real-time performances, and the infinite degrees of freedom of the deformable linear objects. In this paper, we propose a real-time approach for estimating both contact establishment and subsequent changes by leveraging the dependency between the applied and detected contact force on the deformable linear objects. We seamlessly integrate this method into the robot control loop and achieve an adaptive shape control framework which avoids, detects and corrects anomalies automatically. Real-world experiments validate the robustness and effectiveness of our contact estimation approach across various scenarios, significantly increasing the success rate of shape control processes. △ Less

Submitted 30 January, 2024; originally announced January 2024.

ULT-model: Towards a one-legged unified locomotion template model for forward hopping with an upright trunk

Authors: Dennis Ossadnik, Elisabeth Jensen, Sami Haddadin

Abstract: While many advancements have been made in the development of template models for describing upright-trunk locomotion, the majority of the effort has been focused on the stance phase. In this paper, we develop a new compact dynamic model as a first step toward a fully unified locomotion template model (ULT-model) of an upright-trunk forward hopping system, which will also require a unified control… ▽ More While many advancements have been made in the development of template models for describing upright-trunk locomotion, the majority of the effort has been focused on the stance phase. In this paper, we develop a new compact dynamic model as a first step toward a fully unified locomotion template model (ULT-model) of an upright-trunk forward hopping system, which will also require a unified control law in the next step. We demonstrate that all locomotion subfunctions are enabled by adding just a point foot mass and a parallel leg actuator to the well-known trunk SLIP model and that a stable limit cycle can be achieved. This brings us closer toward the ultimate goal of enabling closed-loop dynamics for anchor matching and thus achieving simple, efficient, robust and stable upright-trunk gait control, as observed in biological systems. △ Less

Submitted 16 January, 2024; originally announced January 2024.

Comments: 7 pages, 10 figures

Journal ref: 2021 IEEE International Conference on Robotics and Automation (ICRA 2021), May 31 - June 4, 2021, Xi'an, China, pp. 3040-3046

Nonlinear stiffness allows passive dynamic hopping for one-legged robots with an upright trunk

Authors: Dennis Ossadnik, Elisabeth Jensen, Sami Haddadin

Abstract: Template models are frequently used to simplify the control dynamics for robot hopping or running. Passive limit cycles can emerge for such systems and be exploited for energy-efficient control. A grand challenge in locomotion is trunk stabilization when the hip is offset from the center of mass (CoM). The swing phase plays a major role in this process due to the moment of inertia of the leg; howe… ▽ More Template models are frequently used to simplify the control dynamics for robot hopping or running. Passive limit cycles can emerge for such systems and be exploited for energy-efficient control. A grand challenge in locomotion is trunk stabilization when the hip is offset from the center of mass (CoM). The swing phase plays a major role in this process due to the moment of inertia of the leg; however, many template models ignore the leg mass. In this work, the authors consider a robot hopper model (RHM) with a rigid trunk and leg plus a hip that is displaced from the CoM. It has been previously shown that no passive limit cycle exists for such a model given a linear hip spring. In this work, we show that passive limit cycles can be found when a nonlinear hip spring is used instead. To the authors' knowledge, this is the first time that a passive limit cycle has been found for this type of system. △ Less

Submitted 16 January, 2024; originally announced January 2024.

Comments: 7 pages, 6 figures

Journal ref: 2021 IEEE International Conference on Robotics and Automation (ICRA 2021), May 31 - June 4, 2021, Xi'an, China, pp. 3047 - 3053

Anthropomorphic Grasping with Neural Object Shape Completion

Authors: Diego Hidalgo-Carvajal, Hanzhi Chen, Gemma C. Bettelani, Jaesug Jung, Melissa Zavaglia, Laura Busse, Abdeldjallil Naceri, Stefan Leutenegger, Sami Haddadin

Abstract: The progressive prevalence of robots in human-suited environments has given rise to a myriad of object manipulation techniques, in which dexterity plays a paramount role. It is well-established that humans exhibit extraordinary dexterity when handling objects. Such dexterity seems to derive from a robust understanding of object properties (such as weight, size, and shape), as well as a remarkable… ▽ More The progressive prevalence of robots in human-suited environments has given rise to a myriad of object manipulation techniques, in which dexterity plays a paramount role. It is well-established that humans exhibit extraordinary dexterity when handling objects. Such dexterity seems to derive from a robust understanding of object properties (such as weight, size, and shape), as well as a remarkable capacity to interact with them. Hand postures commonly demonstrate the influence of specific regions on objects that need to be grasped, especially when objects are partially visible. In this work, we leverage human-like object understanding by reconstructing and completing their full geometry from partial observations, and manipulating them using a 7-DoF anthropomorphic robot hand. Our approach has significantly improved the grasping success rates of baselines with only partial reconstruction by nearly 30% and achieved over 150 successful grasps with three different object categories. This demonstrates our approach's consistent ability to predict and execute grasping postures based on the completed object shapes from various directions and positions in real-world scenarios. Our work opens up new possibilities for enhancing robotic applications that require precise grasping and manipulation skills of real-world reconstructed objects. △ Less

Submitted 9 November, 2023; v1 submitted 4 November, 2023; originally announced November 2023.

Comments: Accepted to RA-L 2023

Flexible Informed Trees (FIT*): Adaptive Batch-Size Approach in Informed Sampling-Based Path Planning

Authors: Liding Zhang, Zhenshan Bing, Kejia Chen, Lingyun Chen, Kuanqi Cai, Yu Zhang, Fan Wu, Peter Krumbholz, Zhilin Yuan, Sami Haddadin, Alois Knoll

Abstract: In path planning, anytime almost-surely asymptotically optimal planners dominate the benchmark of sampling-based planners. A notable example is Batch Informed Trees (BIT*), where planners iteratively determine paths to batches of vertices within the exploration area. However, utilizing a consistent batch size is inefficient for initial pathfinding and optimal performance, it relies on effective ta… ▽ More In path planning, anytime almost-surely asymptotically optimal planners dominate the benchmark of sampling-based planners. A notable example is Batch Informed Trees (BIT*), where planners iteratively determine paths to batches of vertices within the exploration area. However, utilizing a consistent batch size is inefficient for initial pathfinding and optimal performance, it relies on effective task allocation. This paper introduces Flexible Informed Trees (FIT*), a sampling-based planner that integrates an adaptive batch-size method to enhance the initial path convergence rate. FIT* employs a flexible approach in adjusting batch sizes dynamically based on the inherent dimension of the configuration spaces and the hypervolume of the n-dimensional hyperellipsoid. By applying dense and sparse sampling strategy, FIT* improves convergence rate while finding successful solutions faster with lower initial solution cost. This method enhances the planner's ability to handle confined, narrow spaces in the initial finding phase and increases batch vertices sampling frequency in the optimization phase. FIT* outperforms existing single-query, sampling-based planners on the tested problems in R^2 to R^8, and was demonstrated on a real-world mobile manipulation task. △ Less

Submitted 2 March, 2024; v1 submitted 19 October, 2023; originally announced October 2023.

Comments: 7 pages,7 figures

LoHoRavens: A Long-Horizon Language-Conditioned Benchmark for Robotic Tabletop Manipulation

Authors: Shengqiang Zhang, Philipp Wicke, Lütfi Kerem Şenel, Luis Figueredo, Abdeldjallil Naceri, Sami Haddadin, Barbara Plank, Hinrich Schütze

Abstract: The convergence of embodied agents and large language models (LLMs) has brought significant advancements to embodied instruction following. Particularly, the strong reasoning capabilities of LLMs make it possible for robots to perform long-horizon tasks without expensive annotated demonstrations. However, public benchmarks for testing the long-horizon reasoning capabilities of language-conditioned… ▽ More The convergence of embodied agents and large language models (LLMs) has brought significant advancements to embodied instruction following. Particularly, the strong reasoning capabilities of LLMs make it possible for robots to perform long-horizon tasks without expensive annotated demonstrations. However, public benchmarks for testing the long-horizon reasoning capabilities of language-conditioned robots in various scenarios are still missing. To fill this gap, this work focuses on the tabletop manipulation task and releases a simulation benchmark, \textit{LoHoRavens}, which covers various long-horizon reasoning aspects spanning color, size, space, arithmetics and reference. Furthermore, there is a key modality bridging problem for long-horizon manipulation tasks with LLMs: how to incorporate the observation feedback during robot execution for the LLM's closed-loop planning, which is however less studied by prior work. We investigate two methods of bridging the modality gap: caption generation and learnable interface for incorporating explicit and implicit observation feedback to the LLM, respectively. These methods serve as the two baselines for our proposed benchmark. Experiments show that both methods struggle to solve some tasks, indicating long-horizon manipulation tasks are still challenging for current popular models. We expect the proposed public benchmark and baselines can help the community develop better models for long-horizon tabletop manipulation tasks. △ Less

Submitted 23 October, 2023; v1 submitted 18 October, 2023; originally announced October 2023.

Comments: 6 pages, 4 figures. The video and code of LoHoRavens are available at https://cisnlp.github.io/lohoravens-webpage/

Care3D: An Active 3D Object Detection Dataset of Real Robotic-Care Environments

Authors: Michael G. Adam, Sebastian Eger, Martin Piccolrovazzi, Maged Iskandar, Joern Vogel, Alexander Dietrich, Seongjien Bien, Jon Skerlj, Abdeldjallil Naceri, Eckehard Steinbach, Alin Albu-Schaeffer, Sami Haddadin, Wolfram Burgard

Abstract: As labor shortage increases in the health sector, the demand for assistive robotics grows. However, the needed test data to develop those robots is scarce, especially for the application of active 3D object detection, where no real data exists at all. This short paper counters this by introducing such an annotated dataset of real environments. The captured environments represent areas which are al… ▽ More As labor shortage increases in the health sector, the demand for assistive robotics grows. However, the needed test data to develop those robots is scarce, especially for the application of active 3D object detection, where no real data exists at all. This short paper counters this by introducing such an annotated dataset of real environments. The captured environments represent areas which are already in use in the field of robotic health care research. We further provide ground truth data within one room, for assessing SLAM algorithms running directly on a health care robot. △ Less

Submitted 9 October, 2023; originally announced October 2023.

A Concise Overview of Safety Aspects in Human-Robot Interaction

Authors: Mazin Hamad, Simone Nertinger, Robin J. Kirschner, Luis Figueredo, Abdeldjallil Naceri, Sami Haddadin

Abstract: As of today, robots exhibit impressive agility but also pose potential hazards to humans using/collaborating with them. Consequently, safety is considered the most paramount factor in human-robot interaction (HRI). This paper presents a multi-layered safety architecture, integrating both physical and cognitive aspects for effective HRI. We outline critical requirements for physical safety layers a… ▽ More As of today, robots exhibit impressive agility but also pose potential hazards to humans using/collaborating with them. Consequently, safety is considered the most paramount factor in human-robot interaction (HRI). This paper presents a multi-layered safety architecture, integrating both physical and cognitive aspects for effective HRI. We outline critical requirements for physical safety layers as service modules that can be arbitrarily queried. Further, we showcase an HRI scheme that addresses human factors and perceived safety as high-level constraints on a validated impact safety paradigm. The aim is to enable safety certification of human-friendly robots across various HRI scenarios. △ Less

Submitted 18 September, 2023; originally announced September 2023.

Comments: Accepted for Human-Friendly Robotics 2023: 16th International Workshop

Optimally Controlling the Timing of Energy Transfer in Elastic Joints: Experimental Validation of the Bi-Stiffness Actuation Concept

Authors: Edmundo Pozo Fortunić, Mehmet C. Yildirim, Dennis Ossadnik, Abdalla Swikir, Saeed Abdolshah, Sami Haddadin

Abstract: Elastic actuation taps into elastic elements' energy storage for dynamic motions beyond rigid actuation. While Series Elastic Actuators (SEA) and Variable Stiffness Actuators (VSA) are highly sophisticated, they do not fully provide control over energy transfer timing. To overcome this problem on the basic system level, the Bi-Stiffness Actuation (BSA) concept was recently proposed. Theoretically,… ▽ More Elastic actuation taps into elastic elements' energy storage for dynamic motions beyond rigid actuation. While Series Elastic Actuators (SEA) and Variable Stiffness Actuators (VSA) are highly sophisticated, they do not fully provide control over energy transfer timing. To overcome this problem on the basic system level, the Bi-Stiffness Actuation (BSA) concept was recently proposed. Theoretically, it allows for full link decoupling, while simultaneously being able to lock the spring in the drive train via a switch-and-hold mechanism. Thus, the user would be in full control of the potential energy storage and release timing. In this work, we introduce an initial proof-of-concept of Bi-Stiffness-Actuation in the form of a 1-DoF physical prototype, which is implemented using a modular testbed. We present a hybrid system model, as well as the mechatronic implementation of the actuator. We corroborate the feasibility of the concept by conducting a series of hardware experiments using an open-loop control signal obtained by trajectory optimization. Here, we compare the performance of the prototype with a comparable SEA implementation. We show that BSA outperforms SEA 1) in terms of maximum velocity at low final times and 2) in terms of the movement strategy itself: The clutch mechanism allows the BSA to generate consistent launch sequences while the SEA has to rely on lengthy and possibly dangerous oscillatory swing-up motions. Furthermore, we demonstrate that providing full control authority over the energy transfer timing and link decoupling allows the user to synchronously release both elastic joint and gravitational energy. This facilitates the optimal exploitation of elastic and gravitational potentials in a synergistic manner. △ Less

Submitted 14 September, 2023; originally announced September 2023.

Comments: 8 pages, 9 figures. Submitted to IEEE Robotics and Automation Letters

Towards Connecting Control to Perception: High-Performance Whole-Body Collision Avoidance Using Control-Compatible Obstacles

Authors: Moritz Eckhoff, Dennis Knobbe, Henning Zwirnmann, Abdalla Swikir, Sami Haddadin

Abstract: One of the most important aspects of autonomous systems is safety. This includes ensuring safe human-robot and safe robot-environment interaction when autonomously performing complex tasks or in collaborative scenarios. Although several methods have been introduced to tackle this, most are unsuitable for real-time applications and require carefully hand-crafted obstacle descriptions. In this work,… ▽ More One of the most important aspects of autonomous systems is safety. This includes ensuring safe human-robot and safe robot-environment interaction when autonomously performing complex tasks or in collaborative scenarios. Although several methods have been introduced to tackle this, most are unsuitable for real-time applications and require carefully hand-crafted obstacle descriptions. In this work, we propose a method combining high-frequency and real-time self and environment collision avoidance of a robotic manipulator with low-frequency, multimodal, and high-resolution environmental perceptions accumulated in a digital twin system. Our method is based on geometric primitives, so-called primitive skeletons. These, in turn, are information-compressed and real-time compatible digital representations of the robot's body and environment, automatically generated from ultra-realistic virtual replicas of the real world provided by the digital twin. Our approach is a key enabler for closing the loop between environment perception and robot control by providing the millisecond real-time control stage with a current and accurate world description, empowering it to react to environmental changes. We evaluate our whole-body collision avoidance on a 9-DOFs robot system through five experiments, demonstrating the functionality and efficiency of our framework. △ Less

Submitted 15 December, 2023; v1 submitted 13 September, 2023; originally announced September 2023.

Comments: Published at 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2023)

Journal ref: 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Detroit, MI, USA, 2023, pp. 2354-2361

Contact-aware Shaping and Maintenance of Deformable Linear Objects With Fixtures

Authors: Kejia Chen, Zhenshan Bing, Fan Wu, Yuan Meng, Andre Kraft, Sami Haddadin, Alois Knoll

Abstract: Studying the manipulation of deformable linear objects has significant practical applications in industry, including car manufacturing, textile production, and electronics automation. However, deformable linear object manipulation poses a significant challenge in developing planning and control algorithms, due to the precise and continuous control required to effectively manipulate the deformable… ▽ More Studying the manipulation of deformable linear objects has significant practical applications in industry, including car manufacturing, textile production, and electronics automation. However, deformable linear object manipulation poses a significant challenge in developing planning and control algorithms, due to the precise and continuous control required to effectively manipulate the deformable nature of these objects. In this paper, we propose a new framework to control and maintain the shape of deformable linear objects with two robot manipulators utilizing environmental contacts. The framework is composed of a shape planning algorithm which automatically generates appropriate positions to place fixtures, and an object-centered skill engine which includes task and motion planning to control the motion and force of both robots based on the object status. The status of the deformable linear object is estimated online utilizing visual as well as force information. The framework manages to handle a cable routing task in real-world experiments with two Panda robots and especially achieves contact-aware and flexible clip fixing with challenging fixtures. △ Less

Submitted 19 July, 2023; originally announced July 2023.

Towards Language-Based Modulation of Assistive Robots through Multimodal Models

Authors: Philipp Wicke, Lüfti Kerem Şenel, Shengqiang Zhang, Luis Figueredo, Abdeldjallil Naceri, Sami Haddadin, Hinrich Schütze

Abstract: In the field of Geriatronics, enabling effective and transparent communication between humans and robots is crucial for enhancing the acceptance and performance of assistive robots. Our early-stage research project investigates the potential of language-based modulation as a means to improve human-robot interaction. We propose to explore real-time modulation during task execution, leveraging langu… ▽ More In the field of Geriatronics, enabling effective and transparent communication between humans and robots is crucial for enhancing the acceptance and performance of assistive robots. Our early-stage research project investigates the potential of language-based modulation as a means to improve human-robot interaction. We propose to explore real-time modulation during task execution, leveraging language cues, visual references, and multimodal inputs. By developing transparent and interpretable methods, we aim to enable robots to adapt and respond to language commands, enhancing their usability and flexibility. Through the exchange of insights and knowledge at the workshop, we seek to gather valuable feedback to advance our research and contribute to the development of interactive robotic systems for Geriatronics and beyond. △ Less

Submitted 27 June, 2023; v1 submitted 26 June, 2023; originally announced June 2023.

Comments: GERIATRONICS SUMMIT 2023

Fast yet predictable braking manoeuvers for real-time robot control

Authors: Mazin Hamad, Jesus Gutierrez-Moreno, Hugo T. M. Kussaba, Nico Mansfeld, Saeed Abdolshah, Abdalla Swikir, Wolfram Burgard, Sami Haddadin

Abstract: This paper proposes a framework for generating fast, smooth and predictable braking manoeuvers for a controlled robot. The proposed framework integrates two approaches to obtain feasible modal limits for designing braking trajectories. The first approach is real-time capable but conservative considering the usage of the available feasible actuator control region, resulting in longer braking times.… ▽ More This paper proposes a framework for generating fast, smooth and predictable braking manoeuvers for a controlled robot. The proposed framework integrates two approaches to obtain feasible modal limits for designing braking trajectories. The first approach is real-time capable but conservative considering the usage of the available feasible actuator control region, resulting in longer braking times. In contrast, the second approach maximizes the used braking control inputs at the cost of requiring more time to evaluate larger, feasible modal limits via optimization. Both approaches allow for predicting the robot's stopping trajectory online. In addition, we also formulated and solved a constrained, nonlinear final-time minimization problem to find optimal torque inputs. The optimal solutions were used as a benchmark to evaluate the performance of the proposed predictable braking framework. A comparative study was compiled in simulation versus a classical optimal controller on a 7-DoF robot arm with only three moving joints. The results verified the effectiveness of our proposed framework and its integrated approaches in achieving fast robot braking manoeuvers with accurate online predictions of the stopping trajectories and distances under various braking settings. △ Less

Submitted 10 June, 2023; originally announced June 2023.

Comments: This work has been accepted to the 22nd IFAC World Congress

Learning optimal controllers: a dynamical motion primitive approach

Authors: Hugo T. M. Kussaba, Abdalla Swikir, Fan Wu, Anastasija Demerdjieva, Gitta Kutyniok, Sami Haddadin

Abstract: Real-time computation of optimal control is a challenging problem and, to solve this difficulty, many frameworks proposed to use learning techniques to learn (possibly sub-optimal) controllers and enable their usage in an online fashion. Among these techniques, the optimal motion framework is a simple, yet powerful technique, that obtained success in many complex real-world applications. The main… ▽ More Real-time computation of optimal control is a challenging problem and, to solve this difficulty, many frameworks proposed to use learning techniques to learn (possibly sub-optimal) controllers and enable their usage in an online fashion. Among these techniques, the optimal motion framework is a simple, yet powerful technique, that obtained success in many complex real-world applications. The main idea of this approach is to take advantage of dynamic motion primitives, a widely used tool in robotics to learn trajectories from demonstrations. While usually these demonstrations come from humans, the optimal motion framework is based on demonstrations coming from optimal solutions, such as the ones obtained by numeric solvers. As usual in many learning techniques, a drawback of this approach is that it is hard to estimate the suboptimality of learned solutions, since finding easily computable and non-trivial upper bounds to the error between an optimal solution and a learned solution is, in general, unfeasible. However, we show in this paper that it is possible to estimate this error for a broad class of problems. Furthermore, we apply this estimation technique to achieve a novel and more efficient sampling scheme to be used within the optimal motion framework, enabling the usage of this framework in some scenarios where the computational resources are limited. △ Less

Submitted 10 June, 2023; originally announced June 2023.

Comments: This work has been accepted to the 22nd IFAC World Congress

Towards Flexible Biolaboratory Automation: Container Taxonomy-Based, 3D-Printed Gripper Fingers

Authors: Henning Zwirnmann, Dennis Knobbe, Utku Culha, Sami Haddadin

Abstract: Automation in the life science research laboratory is a paradigm that has gained increasing relevance in recent years. Current robotic solutions often have a limited scope, which reduces their acceptance and prevents the realization of complex workflows. The transport and manipulation of laboratory supplies with a robot is a particular case where this limitation manifests. In this paper, we deduce… ▽ More Automation in the life science research laboratory is a paradigm that has gained increasing relevance in recent years. Current robotic solutions often have a limited scope, which reduces their acceptance and prevents the realization of complex workflows. The transport and manipulation of laboratory supplies with a robot is a particular case where this limitation manifests. In this paper, we deduce a taxonomy of biolaboratory liquid containers that clarifies the need for a flexible grasping solution. Using the taxonomy as a guideline, we design fingers for a parallel robotic gripper which are developed with a monolithic dual-extrusion 3D print that integrates rigid and soft materials to optimize gripping properties. We design fine-tuned fingertips that provide stable grasps of the containers in question. A simple actuation system and a low weight are maintained by adopting a passive compliant mechanism. The ability to resist chemicals and high temperatures and the integration with a tool exchange system render the fingers usable for daily laboratory use and complex workflows. We present the task suitability of the fingers in experiments that show the wide range of vessels that can be handled as well as their tolerance against displacements and their grasp stability. △ Less

Submitted 15 December, 2023; v1 submitted 7 February, 2023; originally announced February 2023.

Comments: Version of the article that was accepted for publication at the 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

Journal ref: 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Detroit, MI, USA, 2023, pp. 6823-6830

BSA -- Bi-Stiffness Actuation for optimally exploiting intrinsic compliance and inertial coupling effects in elastic joint robots

Authors: Dennis Ossadnik, Mehmet C. Yildirim, Fan Wu, Abdalla Swikir, Hugo T. M. Kussaba, Saeed Abdolshah, Sami Haddadin

Abstract: Compliance in actuation has been exploited to generate highly dynamic maneuvers such as throwing that take advantage of the potential energy stored in joint springs. However, the energy storage and release could not be well-timed yet. On the contrary, for multi-link systems, the natural system dynamics might even work against the actual goal. With the introduction of variable stiffness actuators,… ▽ More Compliance in actuation has been exploited to generate highly dynamic maneuvers such as throwing that take advantage of the potential energy stored in joint springs. However, the energy storage and release could not be well-timed yet. On the contrary, for multi-link systems, the natural system dynamics might even work against the actual goal. With the introduction of variable stiffness actuators, this problem has been partially addressed. With a suitable optimal control strategy, the approximate decoupling of the motor from the link can be achieved to maximize the energy transfer into the distal link prior to launch. However, such continuous stiffness variation is complex and typically leads to oscillatory swing-up motions instead of clear launch sequences. To circumvent this issue, we investigate decoupling for speed maximization with a dedicated novel actuator concept denoted Bi-Stiffness Actuation. With this, it is possible to fully decouple the link from the joint mechanism by a switch-and-hold clutch and simultaneously keep the elastic energy stored. We show that with this novel paradigm, it is not only possible to reach the same optimal performance as with power-equivalent variable stiffness actuation, but even directly control the energy transfer timing. This is a major step forward compared to previous optimal control approaches, which rely on optimizing the full time-series control input. △ Less

Submitted 30 December, 2022; originally announced December 2022.

Comments: Accepted version of article that has been published in 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

Journal ref: 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3536-3543

Informed Circular Fields for Global Reactive Obstacle Avoidance of Robotic Manipulators

Authors: Marvin Becker, Philipp Caspers, Tom Hattendorf, Torsten Lilge, Sami Haddadin, Matthias A. Müller

Abstract: In this paper a global reactive motion planning framework for robotic manipulators in complex dynamic environments is presented. In particular, the circular field predictions (CFP) planner from Becker et al. (2021) is extended to ensure obstacle avoidance of the whole structure of a robotic manipulator. Towards this end, a motion planning framework is developed that leverages global information ab… ▽ More In this paper a global reactive motion planning framework for robotic manipulators in complex dynamic environments is presented. In particular, the circular field predictions (CFP) planner from Becker et al. (2021) is extended to ensure obstacle avoidance of the whole structure of a robotic manipulator. Towards this end, a motion planning framework is developed that leverages global information about promising avoidance directions from arbitrary configuration space motion planners, resulting in improved global trajectories while reactively avoiding dynamic obstacles and decreasing the required computational power. The resulting motion planning framework is tested in multiple simulations with complex and dynamic obstacles and demonstrates great potential compared to existing motion planning approaches. △ Less

Submitted 4 August, 2023; v1 submitted 12 December, 2022; originally announced December 2022.

Comments: Accepted to IFAC World Congress 2023

Motion Planning using Reactive Circular Fields: A 2D Analysis of Collision Avoidance and Goal Convergence

Authors: Marvin Becker, Johannes Köhler, Sami Haddadin, Matthias A. Müller

Abstract: Recently, many reactive trajectory planning approaches were suggested in the literature because of their inherent immediate adaption in the ever more demanding cluttered and unpredictable environments of robotic systems. However, typically those approaches are only locally reactive without considering global path planning and no guarantees for simultaneous collision avoidance and goal convergence… ▽ More Recently, many reactive trajectory planning approaches were suggested in the literature because of their inherent immediate adaption in the ever more demanding cluttered and unpredictable environments of robotic systems. However, typically those approaches are only locally reactive without considering global path planning and no guarantees for simultaneous collision avoidance and goal convergence can be given. In this paper, we study a recently developed circular field (CF)-based motion planner that combines local reactive control with global trajectory generation by adapting an artificial magnetic field such that multiple trajectories around obstacles can be evaluated. In particular, we provide a mathematically rigorous analysis of this planner in a planar environment to ensure safe motion of the controlled robot. Contrary to existing results, the derived collision avoidance analysis covers the entire CF motion planning algorithm including attractive forces for goal convergence and is not limited to a specific choice of the rotation field, i.e., our guarantees are not limited to a specific potentially suboptimal trajectory. Our Lyapunov-type collision avoidance analysis is based on the definition of an (equivalent) two-dimensional auxiliary system, which enables us to provide tight, if and only if conditions for the case of a collision with point obstacles. Furthermore, we show how this analysis naturally extends to multiple obstacles and we specify sufficient conditions for goal convergence. Finally, we provide a challenging simulation scenario with multiple non-convex point cloud obstacles and demonstrate collision avoidance and goal convergence. △ Less

Submitted 3 November, 2023; v1 submitted 28 October, 2022; originally announced October 2022.

Comments: Published in IEEE Transactions on Automatic Control (Early Access)

A Solution to Slosh-free Robot Trajectory Optimization

Authors: Rafael I. Cabral Muchacho, Riddhiman Laha, Luis F. C. Figueredo, Sami Haddadin

Abstract: This paper is about fast slosh free fluid transportation. Existing approaches are either computationally heavy or only suitable for specific robots and container shapes. We model the end effector as a point mass suspended by a spherical pendulum and study the requirements for slosh free motion and the validity of the point mass model. In this approach, slosh free trajectories are generated by cont… ▽ More This paper is about fast slosh free fluid transportation. Existing approaches are either computationally heavy or only suitable for specific robots and container shapes. We model the end effector as a point mass suspended by a spherical pendulum and study the requirements for slosh free motion and the validity of the point mass model. In this approach, slosh free trajectories are generated by controlling the pendulum's pivot and simulating the motion of the point mass. We cast the trajectory optimization problem as a quadratic program; this strategy can be used to obtain valid control inputs. Through simulations and experiments on a 7 DoF Franka Emika Panda robot we validate the effectiveness of the proposed approach. △ Less

Submitted 23 October, 2022; originally announced October 2022.

Comments: IEEE/RSJ IROS 2022

Towards Task-Specific Modular Gripper Fingers: Automatic Production of Fingertip Mechanics

Authors: Johannes Ringwald, Samuel Schneider, Lingyun Chen, Dennis Knobbe, Lars Johannsmeier, Abdalla Swikir, Sami Haddadin

Abstract: The number of sequential tasks a single gripper can perform is significantly limited by its design. In many cases, changing the gripper fingers is required to successfully conduct multiple consecutive tasks. For this reason, several robotic tool change systems have been introduced that allow an automatic changing of the entire end-effector. However, many situations require only the modification or… ▽ More The number of sequential tasks a single gripper can perform is significantly limited by its design. In many cases, changing the gripper fingers is required to successfully conduct multiple consecutive tasks. For this reason, several robotic tool change systems have been introduced that allow an automatic changing of the entire end-effector. However, many situations require only the modification or the change of the fingertip, making the exchange of the entire gripper uneconomic. In this paper, we introduce a paradigm for automatic task-specific fingertip production. The setup used in the proposed framework consists of a production and task execution unit, containing a robotic manipulator, and two 3D printers - autonomously producing the gripper fingers. It also consists of a second manipulator that uses a quick-exchange mechanism to pick up the printed fingertips and evaluates gripping performance. The setup is experimentally validated by conducting automatic production of three different fingertips and executing graspstability tests as well as multiple pick- and insertion tasks, with and without position offsets - using these fingertips. The proposed paradigm, indeed, goes beyond fingertip production and serves as a foundation for a fully automatic fingertip design, production and application pipeline - potentially improving manufacturing flexibility and representing a new production paradigm: tactile 3D manufacturing. △ Less

Submitted 18 October, 2022; originally announced October 2022.

Comments: 8 pages, 9 figures

Human-to-Robot Manipulability Domain Adaptation with Parallel Transport and Manifold-Aware ICP

Authors: Anna Reithmeir, Luis Figueredo, Sami Haddadin

Abstract: Manipulability ellipsoids efficiently capture the human pose and reveal information about the task at hand. Their use in task-dependent robot teaching - particularly their transfer from a teacher to a learner - can advance emulation of human-like motion. Although in recent literature focus is shifted towards manipulability transfer between two robots, the adaptation to the capabilities of the othe… ▽ More Manipulability ellipsoids efficiently capture the human pose and reveal information about the task at hand. Their use in task-dependent robot teaching - particularly their transfer from a teacher to a learner - can advance emulation of human-like motion. Although in recent literature focus is shifted towards manipulability transfer between two robots, the adaptation to the capabilities of the other kinematic system is to date not addressed and research in transfer from human to robot is still in its infancy. This work presents a novel manipulability domain adaptation method for the transfer of manipulability information to the domain of another kinematic system. As manipulability matrices/ellipsoids are symmetric positive-definite (SPD) they can be viewed as points on the Riemannian manifold of SPD matrices. We are the first to address the problem of manipulability transfer from the perspective of point cloud registration. We propose a manifold-aware Iterative Closest Point algorithm (ICP) with parallel transport initialization. Furthermore, we introduce a correspondence matching heuristic for manipulability ellipsoids based on inherent geometric features. We confirm our method in simulation experiments with 2-DoF manipulators as well as 7-DoF models representing the human-arm kinematics. △ Less

Submitted 16 August, 2022; originally announced August 2022.

Comments: 8 pages, 6 figures, to be published in 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

LATTE: LAnguage Trajectory TransformEr

Authors: Arthur Bucker, Luis Figueredo, Sami Haddadin, Ashish Kapoor, Shuang Ma, Sai Vemprala, Rogerio Bonatti

Abstract: Natural language is one of the most intuitive ways to express human intent. However, translating instructions and commands towards robotic motion generation and deployment in the real world is far from being an easy task. The challenge of combining a robot's inherent low-level geometric and kinodynamic constraints with a human's high-level semantic instructions traditionally is solved using task-s… ▽ More Natural language is one of the most intuitive ways to express human intent. However, translating instructions and commands towards robotic motion generation and deployment in the real world is far from being an easy task. The challenge of combining a robot's inherent low-level geometric and kinodynamic constraints with a human's high-level semantic instructions traditionally is solved using task-specific solutions with little generalizability between hardware platforms, often with the use of static sets of target actions and commands. This work instead proposes a flexible language-based framework that allows a user to modify generic robotic trajectories. Our method leverages pre-trained language models (BERT and CLIP) to encode the user's intent and target objects directly from a free-form text input and scene images, fuses geometrical features generated by a transformer encoder network, and finally outputs trajectories using a transformer decoder, without the need of priors related to the task or robot information. We significantly extend our own previous work presented in Bucker et al. by expanding the trajectory parametrization space to 3D and velocity as opposed to just XY movements. In addition, we now train the model to use actual images of the objects in the scene for context (as opposed to textual descriptions), and we evaluate the system in a diverse set of scenarios beyond manipulation, such as aerial and legged robots. Our simulated and real-life experiments demonstrate that our transformer model can successfully follow human intent, modifying the shape and speed of trajectories within multiple environments. Codebase available at: https://github.com/arthurfenderbucker/LaTTe-Language-Trajectory-TransformEr.git △ Less

Submitted 16 September, 2022; v1 submitted 4 August, 2022; originally announced August 2022.

Reshaping Robot Trajectories Using Natural Language Commands: A Study of Multi-Modal Data Alignment Using Transformers

Authors: Arthur Bucker, Luis Figueredo, Sami Haddadin, Ashish Kapoor, Shuang Ma, Rogerio Bonatti

Abstract: Natural language is the most intuitive medium for us to interact with other people when expressing commands and instructions. However, using language is seldom an easy task when humans need to express their intent towards robots, since most of the current language interfaces require rigid templates with a static set of action targets and commands. In this work, we provide a flexible language-based… ▽ More Natural language is the most intuitive medium for us to interact with other people when expressing commands and instructions. However, using language is seldom an easy task when humans need to express their intent towards robots, since most of the current language interfaces require rigid templates with a static set of action targets and commands. In this work, we provide a flexible language-based interface for human-robot collaboration, which allows a user to reshape existing trajectories for an autonomous agent. We take advantage of recent advancements in the field of large language models (BERT and CLIP) to encode the user command, and then combine these features with trajectory information using multi-modal attention transformers. We train the model using imitation learning over a dataset containing robot trajectories modified by language commands, and treat the trajectory generation process as a sequence prediction problem, analogously to how language generation architectures operate. We evaluate the system in multiple simulated trajectory scenarios, and show a significant performance increase of our model over baseline approaches. In addition, our real-world experiments with a robot arm show that users significantly prefer our natural language interface over traditional methods such as kinesthetic teaching or cost-function programming. Our study shows how the field of robotics can take advantage of large pre-trained language models towards creating more intuitive interfaces between robots and machines. Project webpage: https://arthurfenderbucker.github.io/NL_trajectory_reshaper/ △ Less

Submitted 24 March, 2022; originally announced March 2022.

Coordinate Invariant User-Guided Constrained Path Planning with Reactive Rapidly Expanding Plane-Oriented Escaping Trees

Authors: Riddhiman Laha, Ruiai Sun, Wenxi Wu, Dasharadhan Mahalingam, Nilanjan Chakraborty, Luis F. C. Figueredo, Sami Haddadin

Abstract: As collaborative robots move closer to human environments, motion generation and reactive planning strategies that allow for elaborate task execution with minimal easy-to-implement guidance whilst coping with changes in the environment is of paramount importance. In this paper, we present a novel approach for generating real-time motion plans for point-to-point tasks using a single successful huma… ▽ More As collaborative robots move closer to human environments, motion generation and reactive planning strategies that allow for elaborate task execution with minimal easy-to-implement guidance whilst coping with changes in the environment is of paramount importance. In this paper, we present a novel approach for generating real-time motion plans for point-to-point tasks using a single successful human demonstration. Our approach is based on screw linear interpolation,which allows us to respect the underlying geometric constraints that characterize the task and are implicitly present in the demonstration. We also integrate an original reactive collision avoidance approach with our planner. We present extensive experimental results to demonstrate that with our approach,by using a single demonstration of moving one block, we can generate motion plans for complex tasks like stacking multiple blocks (in a dynamic environment). Analogous generalization abilities are also shown for tasks like pouring and loading shelves. For the pouring task, we also show that a demonstration given for one-armed pouring can be used for planning pouring with a dual-armed manipulator of different kinematic structure. △ Less

Submitted 20 March, 2022; originally announced March 2022.

Comments: 8 pages, 7 figures, to be published in IEEE International Conference on Robotics and Automation (ICRA) 2022, Philadelphia (PA) USA

ISO/TS 15066: How Different Interpretations Affect Risk Assessment

Authors: Robin Jeanne Kirschner, Nico Mansfeld, Saeed Abdolshah, Sami Haddadin

Abstract: The current technical specification ISO/TS15066:2016(E) for safe human-robot interaction contains logically conflicting definitions for the contact between human and robot. This may result in different interpretations for the contact classification and thus no unique outcome can be expected, which may even cause a risk to the human. In previous work, we showed a first set of implications. This pap… ▽ More The current technical specification ISO/TS15066:2016(E) for safe human-robot interaction contains logically conflicting definitions for the contact between human and robot. This may result in different interpretations for the contact classification and thus no unique outcome can be expected, which may even cause a risk to the human. In previous work, we showed a first set of implications. This paper addresses the possible interpretations of a collision scenario as a result of the varying interpretations for a risk assessment. With an experiment including four commercially available robot systems we demonstrate the procedure of the risk assessment following the different interpretations of the TS. The results indicate possible incorrect use of the technical specification, which we believe needs to be resolved in future revisions. For this, we suggest tools in form of a decision tree and constrained collision force maps, which enable a simple, unambiguous risk assessment for HRI. △ Less

Submitted 5 March, 2022; originally announced March 2022.

Towards Remote Robotic Competitions: An Internet-Connected Task Board and Dashboard

Authors: Peter So, Jonas Wittmann, Patrick Ruhkamp, Andriy Sarabakha, Sami Haddadin

Abstract: In this work we present a platform to assess robot platform skills using an internet-of-things (IoT) task board device to aggregate performances across remote sites. We demonstrate a concept for a modular, scale-able device and web dashboard enabling remote competitions as an alternative to in-person robot competitions. We share data from nine robot platforms located across four continents in thre… ▽ More In this work we present a platform to assess robot platform skills using an internet-of-things (IoT) task board device to aggregate performances across remote sites. We demonstrate a concept for a modular, scale-able device and web dashboard enabling remote competitions as an alternative to in-person robot competitions. We share data from nine robot platforms located across four continents in three manipulation task categories of object localization, object insertion, and component disassembly through an organized international robot competition - the Robothon Grand Challenge. This paper discusses the design of an electronic task board, the strategies implemented by the top-performing teams and compares their results with a benchmark solution to the presented task board. Through this platform, we demonstrate fully remote, online competitions can generate innovative robotic solutions and tested a tool for measuring remote performances. Using the open-sourced task board code and design files, the reader can reproduce the benchmark solution or configure the platform for their own use case and share their results transparently without transporting their robot platform. △ Less

Submitted 24 January, 2022; originally announced January 2022.

Comments: This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible

Learning Robotic Manipulation Skills Using an Adaptive Force-Impedance Action Space

Authors: Maximilian Ulmer, Elie Aljalbout, Sascha Schwarz, Sami Haddadin

Abstract: Intelligent agents must be able to think fast and slow to perform elaborate manipulation tasks. Reinforcement Learning (RL) has led to many promising results on a range of challenging decision-making tasks. However, in real-world robotics, these methods still struggle, as they require large amounts of expensive interactions and have slow feedback loops. On the other hand, fast human-like adaptive… ▽ More Intelligent agents must be able to think fast and slow to perform elaborate manipulation tasks. Reinforcement Learning (RL) has led to many promising results on a range of challenging decision-making tasks. However, in real-world robotics, these methods still struggle, as they require large amounts of expensive interactions and have slow feedback loops. On the other hand, fast human-like adaptive control methods can optimize complex robotic interactions, yet fail to integrate multimodal feedback needed for unstructured tasks. In this work, we propose to factor the learning problem in a hierarchical learning and adaption architecture to get the best of both worlds. The framework consists of two components, a slow reinforcement learning policy optimizing the task strategy given multimodal observations, and a fast, real-time adaptive control policy continuously optimizing the motion, stability, and effort of the manipulator. We combine these components through a bio-inspired action space that we call AFORCE. We demonstrate the new action space on a contact-rich manipulation task on real hardware and evaluate its performance on three simulated manipulation tasks. Our experiments show that AFORCE drastically improves sample efficiency while reducing energy consumption and improving safety. △ Less

Submitted 20 October, 2021; v1 submitted 19 October, 2021; originally announced October 2021.

ACM Class: I.2.6; I.2.8; I.2.9

Expectable Motion Unit: Avoiding Hazards From Human Involuntary Motions in Human-Robot Interaction

Authors: Robin Jeanne Kirschner, Henning Mayer, Lisa Burr, Nico Mansfeld, Saeed Abdolshah, Sami Haddadin

Abstract: In robotics, many control and planning schemes have been developed to ensure human physical safety in human-robot interaction. The human psychological state and the expectation towards the robot, however, are typically neglected. Even if the robot behaviour is regarded as biomechanically safe, humans may still react with a rapid involuntary motion (IM) caused by a startle or surprise. Such sudden,… ▽ More In robotics, many control and planning schemes have been developed to ensure human physical safety in human-robot interaction. The human psychological state and the expectation towards the robot, however, are typically neglected. Even if the robot behaviour is regarded as biomechanically safe, humans may still react with a rapid involuntary motion (IM) caused by a startle or surprise. Such sudden, uncontrolled motions can jeopardize safety and should be prevented by any means. In this letter, we propose the Expectable Motion Unit (EMU), which ensures that a certain probability of IM occurrence is not exceeded in a typical HRI setting. Based on a model of IM occurrence generated through an experiment with 29 participants, we establish the mapping between robot velocity, robot-human distance, and the relative frequency of IM occurrence. This mapping is processed towards a real-time capable robot motion generator that limits the robot velocity during task execution if necessary. The EMU is combined in a holistic safety framework that integrates both the physical and psychological safety knowledge. A validation experiment showed that the EMU successfully avoids human IM in five out of six cases. △ Less

Submitted 18 August, 2024; v1 submitted 15 September, 2021; originally announced September 2021.

Journal ref: in IEEE Robotics and Automation Letters, vol. 7, no. 2, pp. 2993-3000, April 2022

SyReNets: Symbolic Residual Neural Networks

Authors: Carlos Magno C. O. Valle, Sami Haddadin

Abstract: Despite successful seminal works on passive systems in the literature, learning free-form physical laws for controlled dynamical systems given experimental data is still an open problem. For decades, symbolic mathematical equations and system identification were the golden standards. Unfortunately, a set of assumptions about the properties of the underlying system is required, which makes the mode… ▽ More Despite successful seminal works on passive systems in the literature, learning free-form physical laws for controlled dynamical systems given experimental data is still an open problem. For decades, symbolic mathematical equations and system identification were the golden standards. Unfortunately, a set of assumptions about the properties of the underlying system is required, which makes the model very rigid and unable to adapt to unforeseen changes in the physical system. Neural networks, on the other hand, are known universal function approximators but are prone to over-fit, limited accuracy, and bias problems, which makes them alone unreliable candidates for such tasks. In this paper, we propose SyReNets, an approach that leverages neural networks for learning symbolic relations to accurately describe dynamic physical systems from data. It explores a sequence of symbolic layers that build, in a residual manner, mathematical relations that describes a given desired output from input variables. We apply it to learn the symbolic equation that describes the Lagrangian of a given physical system. We do this by only observing random samples of position, velocity, and acceleration as input and torque as output. Therefore, using the Lagrangian as a latent representation from which we derive torque using the Euler-Lagrange equations. The approach is evaluated using a simulated controlled double pendulum and compared with neural networks, genetic programming, and traditional system identification. The results demonstrate that, compared to neural networks and genetic programming, SyReNets converges to representations that are more accurate and precise throughout the state space. Despite having slower convergence than traditional system identification, similar to neural networks, the approach remains flexible enough to adapt to an unforeseen change in the physical system structure. △ Less

Submitted 29 May, 2021; originally announced May 2021.

Comments: 11 pages, 3 figures, 2 table

Learning Vision-based Reactive Policies for Obstacle Avoidance

Authors: Elie Aljalbout, Ji Chen, Konstantin Ritt, Maximilian Ulmer, Sami Haddadin

Abstract: In this paper, we address the problem of vision-based obstacle avoidance for robotic manipulators. This topic poses challenges for both perception and motion generation. While most work in the field aims at improving one of those aspects, we provide a unified framework for approaching this problem. The main goal of this framework is to connect perception and motion by identifying the relationship… ▽ More In this paper, we address the problem of vision-based obstacle avoidance for robotic manipulators. This topic poses challenges for both perception and motion generation. While most work in the field aims at improving one of those aspects, we provide a unified framework for approaching this problem. The main goal of this framework is to connect perception and motion by identifying the relationship between the visual input and the corresponding motion representation. To this end, we propose a method for learning reactive obstacle avoidance policies. We evaluate our method on goal-reaching tasks for single and multiple obstacles scenarios. We show the ability of the proposed method to efficiently learn stable obstacle avoidance strategies at a high success rate, while maintaining closed-loop responsiveness required for critical applications like human-robot interaction. △ Less

Submitted 30 October, 2020; originally announced October 2020.

Comments: Accepted for publication at CoRL 2020

ACM Class: I.2.9; I.2.6; I.2.10; I.5.0; J.2

How to Make Deep RL Work in Practice

Authors: Nirnai Rao, Elie Aljalbout, Axel Sauer, Sami Haddadin

Abstract: In recent years, challenging control problems became solvable with deep reinforcement learning (RL). To be able to use RL for large-scale real-world applications, a certain degree of reliability in their performance is necessary. Reported results of state-of-the-art algorithms are often difficult to reproduce. One reason for this is that certain implementation details influence the performance sig… ▽ More In recent years, challenging control problems became solvable with deep reinforcement learning (RL). To be able to use RL for large-scale real-world applications, a certain degree of reliability in their performance is necessary. Reported results of state-of-the-art algorithms are often difficult to reproduce. One reason for this is that certain implementation details influence the performance significantly. Commonly, these details are not highlighted as important techniques to achieve state-of-the-art performance. Additionally, techniques from supervised learning are often used by default but influence the algorithms in a reinforcement learning setting in different and not well-understood ways. In this paper, we investigate the influence of certain initialization, input normalization, and adaptive learning techniques on the performance of state-of-the-art RL algorithms. We make suggestions which of those techniques to use by default and highlight areas that could benefit from a solution specifically tailored to RL. △ Less

Submitted 10 November, 2020; v1 submitted 25 October, 2020; originally announced October 2020.

Comments: Published as a workshop paper at the Deep RL workshop, NeurIPS 2020

Tracking Holistic Object Representations

Authors: Axel Sauer, Elie Aljalbout, Sami Haddadin

Abstract: Recent advances in visual tracking are based on siamese feature extractors and template matching. For this category of trackers, latest research focuses on better feature embeddings and similarity measures. In this work, we focus on building holistic object representations for tracking. We propose a framework that is designed to be used on top of previous trackers without any need for further trai… ▽ More Recent advances in visual tracking are based on siamese feature extractors and template matching. For this category of trackers, latest research focuses on better feature embeddings and similarity measures. In this work, we focus on building holistic object representations for tracking. We propose a framework that is designed to be used on top of previous trackers without any need for further training of the siamese network. The framework leverages the idea of obtaining additional object templates during the tracking process. Since the number of stored templates is limited, our method only keeps the most diverse ones. We achieve this by providing a new diversity measure in the space of siamese features. The obtained representation contains information beyond the ground truth object location provided to the system. It is then useful for tracking itself but also for further tasks which require a visual understanding of objects. Strong empirical results on tracking benchmarks indicate that our method can improve the performance and robustness of the underlying trackers while barely reducing their speed. In addition, our method is able to match current state-of-the-art results, while using a simpler and older network architecture and running three times faster. △ Less

Submitted 6 August, 2019; v1 submitted 21 July, 2019; originally announced July 2019.

Comments: Accepted for oral presentation at BMVC 2019

MSC Class: 68T45; 62H30; 62H35; 51N99; 68T05; 68T20; 68T40; 68W25; 15B99 ACM Class: I.4.10; I.2.10; I.2.6; I.2.9

Simultaneous Contact and Aerodynamic Force Estimation (s-CAFE) for Aerial Robots

Authors: Teodor Tomić, Philipp Lutz, Korbinian Schmid, Andrew Mathers, Sami Haddadin

Abstract: In this paper, we consider the problem of multirotor flying robots physically interacting with the environment under wind influence. The result are the first algorithms for simultaneous online estimation of contact and aerodynamic wrenches acting on the robot based on real-world data, without the need for dedicated sensors. For this purpose, we investigate two model-based techniques for discrimina… ▽ More In this paper, we consider the problem of multirotor flying robots physically interacting with the environment under wind influence. The result are the first algorithms for simultaneous online estimation of contact and aerodynamic wrenches acting on the robot based on real-world data, without the need for dedicated sensors. For this purpose, we investigate two model-based techniques for discriminating between aerodynamic and interaction forces. The first technique is based on aerodynamic and contact torque models, and uses the external force to estimate wind speed. Contacts are then detected based on the residual between estimated external torque and expected (modeled) aerodynamic torque. Upon detecting contact, wind speed is assumed to change very slowly. From the estimated interaction wrench, we are also able to determine the contact location. This is embedded into a particle filter framework to further improve contact location estimation. The second algorithm uses the propeller aerodynamic power and angular speed as measured by the speed controllers to obtain an estimate of the airspeed. An aerodynamics model is then used to determine the aerodynamic wrench. Both methods rely on accurate aerodynamics models. Therefore, we evaluate data-driven and physics based models as well as offline system identification for flying robots. For obtaining ground truth data we performed autonomous flights in a 3D wind tunnel. Using this data, aerodynamic model selection, parameter identification, and discrimination between aerodynamic and contact forces could be done. Finally, the developed methods could serve as useful estimators for interaction control schemes with simultaneous compensation of wind disturbances. △ Less

Submitted 30 October, 2018; originally announced October 2018.