-
Autonomous Behavior Planning For Humanoid Loco-manipulation Through Grounded Language Model
Authors:
Jin Wang,
Arturo Laurenzi,
Nikos Tsagarakis
Abstract:
Enabling humanoid robots to perform autonomously loco-manipulation in unstructured environments is crucial and highly challenging for achieving embodied intelligence. This involves robots being able to plan their actions and behaviors in long-horizon tasks while using multi-modality to perceive deviations between task execution and high-level planning. Recently, large language models (LLMs) have d…
▽ More
Enabling humanoid robots to perform autonomously loco-manipulation in unstructured environments is crucial and highly challenging for achieving embodied intelligence. This involves robots being able to plan their actions and behaviors in long-horizon tasks while using multi-modality to perceive deviations between task execution and high-level planning. Recently, large language models (LLMs) have demonstrated powerful planning and reasoning capabilities for comprehension and processing of semantic information through robot control tasks, as well as the usability of analytical judgment and decision-making for multi-modal inputs. To leverage the power of LLMs towards humanoid loco-manipulation, we propose a novel language-model based framework that enables robots to autonomously plan behaviors and low-level execution under given textual instructions, while observing and correcting failures that may occur during task execution. To systematically evaluate this framework in grounding LLMs, we created the robot 'action' and 'sensing' behavior library for task planning, and conducted mobile manipulation tasks and experiments in both simulated and real environments using the CENTAURO robot, and verified the effectiveness and application of this approach in robotic tasks with autonomous behavioral planning.
△ Less
Submitted 15 August, 2024;
originally announced August 2024.
-
HYPERmotion: Learning Hybrid Behavior Planning for Autonomous Loco-manipulation
Authors:
Jin Wang,
Rui Dai,
Weijie Wang,
Luca Rossini,
Francesco Ruscelli,
Nikos Tsagarakis
Abstract:
Enabling robots to autonomously perform hybrid motions in diverse environments can be beneficial for long-horizon tasks such as material handling, household chores, and work assistance. This requires extensive exploitation of intrinsic motion capabilities, extraction of affordances from rich environmental information, and planning of physical interaction behaviors. Despite recent progress has demo…
▽ More
Enabling robots to autonomously perform hybrid motions in diverse environments can be beneficial for long-horizon tasks such as material handling, household chores, and work assistance. This requires extensive exploitation of intrinsic motion capabilities, extraction of affordances from rich environmental information, and planning of physical interaction behaviors. Despite recent progress has demonstrated impressive humanoid whole-body control abilities, they struggle to achieve versatility and adaptability for new tasks. In this work, we propose HYPERmotion, a framework that learns, selects and plans behaviors based on tasks in different scenarios. We combine reinforcement learning with whole-body optimization to generate motion for 38 actuated joints and create a motion library to store the learned skills. We apply the planning and reasoning features of the large language models (LLMs) to complex loco-manipulation tasks, constructing a hierarchical task graph that comprises a series of primitive behaviors to bridge lower-level execution with higher-level planning. By leveraging the interaction of distilled spatial geometry and 2D observation with a visual language model (VLM) to ground knowledge into a robotic morphology selector to choose appropriate actions in single- or dual-arm, legged or wheeled locomotion. Experiments in simulation and real-world show that learned motions can efficiently adapt to new tasks, demonstrating high autonomy from free-text commands in unstructured scenes. Videos and website: hy-motion.github.io/
△ Less
Submitted 20 June, 2024;
originally announced June 2024.
-
Task-Driven Computational Framework for Simultaneously Optimizing Design and Mounted Pose of Modular Reconfigurable Manipulators
Authors:
Maolin Lei,
Edoardo Romiti,
Arturo Laurenz,
Nikos G. Tsagarakis
Abstract:
Modular reconfigurable manipulators enable quick adaptation and versatility to address different application environments and tailor to the specific requirements of the tasks. Task performance significantly depends on the manipulator's mounted pose and morphology design, therefore posing the need of methodologies for selecting suitable modular robot configurations and mounted pose that can address…
▽ More
Modular reconfigurable manipulators enable quick adaptation and versatility to address different application environments and tailor to the specific requirements of the tasks. Task performance significantly depends on the manipulator's mounted pose and morphology design, therefore posing the need of methodologies for selecting suitable modular robot configurations and mounted pose that can address the specific task requirements and required performance. Morphological changes in modular robots can be derived through a discrete optimization process involving the selective addition or removal of modules. In contrast, the adjustment of the mounted pose operates within a continuous space, allowing for smooth and precise alterations in both orientation and position. This work introduces a computational framework that simultaneously optimizes modular manipulators' mounted pose and morphology. The core of the work is that we design a mapping function that \textit{implicitly} captures the morphological state of manipulators in the continuous space. This transformation function unifies the optimization of mounted pose and morphology within a continuous space. Furthermore, our optimization framework incorporates a array of performance metrics, such as minimum joint effort and maximum manipulability, and considerations for trajectory execution error and physical and safety constraints. To highlight our method's benefits, we compare it with previous methods that framed such problem as a combinatorial optimization problem and demonstrate its practicality in selecting the modular robot configuration for executing a drilling task with the CONCERT modular robotic platform.
△ Less
Submitted 3 May, 2024;
originally announced May 2024.
-
Semantic-based Loco-Manipulation for Human-Robot Collaboration in Industrial Environments
Authors:
Federico Rollo,
Gennaro Raiola,
Nikolaos Tsagarakis,
Marco Roveri,
Enrico Mingo Hoffman,
Arash Ajoudani
Abstract:
Robots with a high level of autonomy are increasingly requested by smart industries. A way to reduce the workers' stress and effort is to optimize the working environment by taking advantage of autonomous collaborative robots. A typical task for Human-Robot Collaboration (HRC) which improves the working setup in an industrial environment is the \textit{"bring me an object please"} where the user a…
▽ More
Robots with a high level of autonomy are increasingly requested by smart industries. A way to reduce the workers' stress and effort is to optimize the working environment by taking advantage of autonomous collaborative robots. A typical task for Human-Robot Collaboration (HRC) which improves the working setup in an industrial environment is the \textit{"bring me an object please"} where the user asks the collaborator to search for an object while he/she is focused on something else. As often happens, science fiction is ahead of the times, indeed, in the \textit{Iron Man} movie, the robot \textit{Dum-E} helps its creator, \textit{Tony Stark}, to create its famous armours. The ability of the robot to comprehend the semantics of the environment and engage with it is valuable for the human execution of more intricate tasks. In this work, we reproduce this operation to enable a mobile robot with manipulation and grasping capabilities to leverage its geometric and semantic understanding of the environment for the execution of the \textit{Bring Me} action, thereby assisting a worker autonomously. Results are provided to validate the proposed workflow in a simulated environment populated with objects and people. This framework aims to take a step forward in assistive robotics autonomy for industries and domestic environments.
△ Less
Submitted 22 December, 2023;
originally announced December 2023.
-
FollowMe: a Robust Person Following Framework Based on Re-Identification and Gestures
Authors:
Federico Rollo,
Andrea Zunino,
Gennaro Raiola,
Fabio Amadio,
Arash Ajoudani,
Nikolaos Tsagarakis
Abstract:
Human-robot interaction (HRI) has become a crucial enabler in houses and industries for facilitating operational flexibility. When it comes to mobile collaborative robots, this flexibility can be further increased due to the autonomous mobility and navigation capacity of the robotic agents, expanding their workspace and consequently, the personalizable assistance they can provide to the human oper…
▽ More
Human-robot interaction (HRI) has become a crucial enabler in houses and industries for facilitating operational flexibility. When it comes to mobile collaborative robots, this flexibility can be further increased due to the autonomous mobility and navigation capacity of the robotic agents, expanding their workspace and consequently, the personalizable assistance they can provide to the human operators. This however requires that the robot is capable of detecting and identifying the human counterpart in all stages of the collaborative task, and in particular while following a human in crowded workplaces. To respond to this need, we developed a unified perception and navigation framework, which enables the robot to identify and follow a target person using a combination of visual Re-Identification (Re-ID), hand gestures detection, and collision-free navigation. The Re-ID module can autonomously learn the features of a target person and use the acquired knowledge to visually re-identify the target. The navigation stack is used to follow the target avoiding obstacles and other individuals in the environment. Experiments are conducted with few subjects in a laboratory setting where some unknown dynamic obstacles are introduced.
△ Less
Submitted 21 November, 2023;
originally announced November 2023.
-
CARPE-ID: Continuously Adaptable Re-identification for Personalized Robot Assistance
Authors:
Federico Rollo,
Andrea Zunino,
Nikolaos Tsagarakis,
Enrico Mingo Hoffman,
Arash Ajoudani
Abstract:
In today's Human-Robot Interaction (HRI) scenarios, a prevailing tendency exists to assume that the robot shall cooperate with the closest individual or that the scene involves merely a singular human actor. However, in realistic scenarios, such as shop floor operations, such an assumption may not hold and personalized target recognition by the robot in crowded environments is required. To fulfil…
▽ More
In today's Human-Robot Interaction (HRI) scenarios, a prevailing tendency exists to assume that the robot shall cooperate with the closest individual or that the scene involves merely a singular human actor. However, in realistic scenarios, such as shop floor operations, such an assumption may not hold and personalized target recognition by the robot in crowded environments is required. To fulfil this requirement, in this work, we propose a person re-identification module based on continual visual adaptation techniques that ensure the robot's seamless cooperation with the appropriate individual even subject to varying visual appearances or partial or complete occlusions. We test the framework singularly using recorded videos in a laboratory environment and an HRI scenario, i.e., a person-following task by a mobile robot. The targets are asked to change their appearance during tracking and to disappear from the camera field of view to test the challenging cases of occlusion and outfit variations. We compare our framework with one of the state-of-the-art Multi-Object Tracking (MOT) methods and the results show that the CARPE-ID can accurately track each selected target throughout the experiments in all the cases (except two limit cases). At the same time, the s-o-t-a MOT has a mean of 4 tracking errors for each video.
△ Less
Submitted 31 January, 2024; v1 submitted 30 October, 2023;
originally announced October 2023.
-
Whole-body MPC for highly redundant legged manipulators: experimental evaluation with a 37 DoF dual-arm quadruped
Authors:
Ioannis Dadiotis,
Arturo Laurenzi,
Nikos Tsagarakis
Abstract:
Recent progress in legged locomotion has rendered quadruped manipulators a promising solution for performing tasks that require both mobility and manipulation (loco-manipulation). In the real world, task specifications and/or environment constraints may require the quadruped manipulator to be equipped with high redundancy as well as whole-body motion coordination capabilities. This work presents a…
▽ More
Recent progress in legged locomotion has rendered quadruped manipulators a promising solution for performing tasks that require both mobility and manipulation (loco-manipulation). In the real world, task specifications and/or environment constraints may require the quadruped manipulator to be equipped with high redundancy as well as whole-body motion coordination capabilities. This work presents an experimental evaluation of a whole-body Model Predictive Control (MPC) framework achieving real-time performance on a dual-arm quadruped platform consisting of 37 actuated joints. To the best of our knowledge this is the legged manipulator with the highest number of joints to be controlled with real-time whole-body MPC so far. The computational efficiency of the MPC while considering the full robot kinematics and the centroidal dynamics model builds upon an open-source DDP-variant solver and a state-of-the-art optimal control problem formulation. Differently from previous works on quadruped manipulators, the MPC is directly interfaced with the low-level joint impedance controllers without the need of designing an instantaneous whole-body controller. The feasibility on the real hardware is showcased using the CENTAURO platform for the challenging task of picking a heavy object from the ground. Dynamic stepping (trotting) is also showcased for first time with this robot. The results highlight the potential of replanning with whole-body information in a predictive control loop.
△ Less
Submitted 25 October, 2023; v1 submitted 4 October, 2023;
originally announced October 2023.
-
In-Hand Re-grasp Manipulation with Passive Dynamic Actions via Imitation Learning
Authors:
Dehao Wei,
Guokang Sun,
Zeyu Ren,
Shuang Li,
Zhufeng Shao,
Xiang Li,
Nikos Tsagarakis,
Shaohua Ma
Abstract:
Re-grasp manipulation leverages on ergonomic tools to assist humans in accomplishing diverse tasks. In certain scenarios, humans often employ external forces to effortlessly and precisely re-grasp tools like a hammer. Previous development on controllers for in-grasp sliding motion using passive dynamic actions (e.g.,gravity) relies on apprehension of finger-object contact information, and requires…
▽ More
Re-grasp manipulation leverages on ergonomic tools to assist humans in accomplishing diverse tasks. In certain scenarios, humans often employ external forces to effortlessly and precisely re-grasp tools like a hammer. Previous development on controllers for in-grasp sliding motion using passive dynamic actions (e.g.,gravity) relies on apprehension of finger-object contact information, and requires customized design for individual objects with varied geometry and weight distribution. It limits their adaptability to diverse objects. In this paper, we propose an end-to-end sliding motion controller based on imitation learning (IL) that necessitates minimal prior knowledge of object mechanics, relying solely on object position information. To expedite training convergence, we utilize a data glove to collect expert data trajectories and train the policy through Generative Adversarial Imitation Learning (GAIL). Simulation results demonstrate the controller's versatility in performing in-hand sliding tasks with objects of varying friction coefficients, geometric shapes, and masses. By migrating to a physical system using visual position estimation, the controller demonstrated an average success rate of 86%, surpassing the baseline algorithm's success rate of 35% of Behavior Cloning(BC) and 20% of Proximal Policy Optimization (PPO).
△ Less
Submitted 27 September, 2023;
originally announced September 2023.
-
Artifacts Mapping: Multi-Modal Semantic Mapping for Object Detection and 3D Localization
Authors:
Federico Rollo,
Gennaro Raiola,
Andrea Zunino,
Nikolaos Tsagarakis,
Arash Ajoudani
Abstract:
Geometric navigation is nowadays a well-established field of robotics and the research focus is shifting towards higher-level scene understanding, such as Semantic Mapping. When a robot needs to interact with its environment, it must be able to comprehend the contextual information of its surroundings. This work focuses on classifying and localising objects within a map, which is under constructio…
▽ More
Geometric navigation is nowadays a well-established field of robotics and the research focus is shifting towards higher-level scene understanding, such as Semantic Mapping. When a robot needs to interact with its environment, it must be able to comprehend the contextual information of its surroundings. This work focuses on classifying and localising objects within a map, which is under construction (SLAM) or already built. To further explore this direction, we propose a framework that can autonomously detect and localize predefined objects in a known environment using a multi-modal sensor fusion approach (combining RGB and depth data from an RGB-D camera and a lidar). The framework consists of three key elements: understanding the environment through RGB data, estimating depth through multi-modal sensor fusion, and managing artifacts (i.e., filtering and stabilizing measurements). The experiments show that the proposed framework can accurately detect 98% of the objects in the real sample environment, without post-processing, while 85% and 80% of the objects were mapped using the single RGBD camera or RGB + lidar setup respectively. The comparison with single-sensor (camera or lidar) experiments is performed to show that sensor fusion allows the robot to accurately detect near and far obstacles, which would have been noisy or imprecise in a purely visual or laser-based approach.
△ Less
Submitted 21 November, 2023; v1 submitted 3 July, 2023;
originally announced July 2023.
-
Design and Validation of a Multi-Arm Relocatable Manipulator for Space Applications
Authors:
Enrico Mingo Hoffman,
Arturo Laurenzi,
Francesco Ruscelli,
Luca Rossini,
Lorenzo Baccelliere,
Davide Antonucci,
Alessio Margan,
Paolo Guria,
Marco Migliorini,
Stefano Cordasco,
Gennaro Raiola,
Luca Muratore,
Joaquín Estremera Rodrigo,
Andrea Rusconi,
Guido Sangiovanni,
Nikos G. Tsagarakis
Abstract:
This work presents the computational design and validation of the Multi-Arm Relocatable Manipulator (MARM), a three-limb robot for space applications, with particular reference to the MIRROR (i.e., the Multi-arm Installation Robot for Readying ORUs and Reflectors) use-case scenario as proposed by the European Space Agency. A holistic computational design and validation pipeline is proposed, with t…
▽ More
This work presents the computational design and validation of the Multi-Arm Relocatable Manipulator (MARM), a three-limb robot for space applications, with particular reference to the MIRROR (i.e., the Multi-arm Installation Robot for Readying ORUs and Reflectors) use-case scenario as proposed by the European Space Agency. A holistic computational design and validation pipeline is proposed, with the aim of comparing different limb designs, as well as ensuring that valid limb candidates enable MARM to perform the complex loco-manipulation tasks required. Motivated by the task complexity in terms of kinematic reachability, (self)-collision avoidance, contact wrench limits, and motor torque limits affecting Earth experiments, this work leverages on multiple state-of-art planning and control approaches to aid the robot design and validation. These include sampling-based planning on manifolds, non-linear trajectory optimization, and quadratic programs for inverse dynamics computations with constraints. Finally, we present the attained MARM design and conduct preliminary tests for hardware validation through a set of lab experiments.
△ Less
Submitted 24 January, 2023;
originally announced January 2023.
-
Trajectory Optimization for Quadruped Mobile Manipulators that Carry Heavy Payload
Authors:
Ioannis Dadiotis,
Arturo Laurenzi,
Nikos Tsagarakis
Abstract:
This paper presents a simplified model-based trajectory optimization (TO) formulation for motion planning on quadruped mobile manipulators that carry heavy payload of known mass. The proposed payload-aware formulation simultaneously plans locomotion, payload manipulation and considers both robot and payload model dynamics while remaining computationally efficient. At the presence of heavy payload,…
▽ More
This paper presents a simplified model-based trajectory optimization (TO) formulation for motion planning on quadruped mobile manipulators that carry heavy payload of known mass. The proposed payload-aware formulation simultaneously plans locomotion, payload manipulation and considers both robot and payload model dynamics while remaining computationally efficient. At the presence of heavy payload, the approach exhibits reduced leg outstretching (thus increased manipulability) in kinematically demanding motions due to the contribution of payload manipulation in the optimization. The framework's computational efficiency and performance is validated through a number of simulation and experimental studies with the bi-manual quadruped CENTAURO robot carrying on its arms a payload that exceeds 15 % of its mass and traversing non-flat terrain.
△ Less
Submitted 13 October, 2022;
originally announced October 2022.
-
Prototyping fast and agile motions for legged robots with Horizon
Authors:
Francesco Ruscelli,
Arturo Laurenzi,
Nikos G. Tsagarakis,
Enrico Mingo Hoffman
Abstract:
For legged robots to perform agile, highly dynamic and contact-rich motions, whole-body trajectories computation of under-actuated complex systems subject to non-linear dynamics is required. In this work, we present hands-on applications of Horizon, a novel open-source framework for trajectory optimization tailored to robotic systems, that provides a collection of tools to simplify dynamic motion…
▽ More
For legged robots to perform agile, highly dynamic and contact-rich motions, whole-body trajectories computation of under-actuated complex systems subject to non-linear dynamics is required. In this work, we present hands-on applications of Horizon, a novel open-source framework for trajectory optimization tailored to robotic systems, that provides a collection of tools to simplify dynamic motion generation. Horizon was tested on a broad range of behaviours involving several robotic platforms: we introduce its building blocks and describe the complete procedure to generate three complex motions using its intuitive and straightforward API.
△ Less
Submitted 17 June, 2022;
originally announced June 2022.
-
Loco-Manipulation Planning for Legged Robots: Offline and Online Strategies
Authors:
Luca Rossini,
Paolo Ferrari,
Francesco Ruscelli,
Arturo Laurenzi,
Nikos G. Tsagarakis,
Enrico Mingo Hoffman
Abstract:
The deployment of robots within realistic environments requires the capability to plan and refine the loco-manipulation trajectories on the fly to avoid unexpected interactions with a dynamic environment. This extended abstract provides a pipeline to offline plan a configuration space global trajectory based on a randomized strategy, and to online locally refine it depending on any change of the d…
▽ More
The deployment of robots within realistic environments requires the capability to plan and refine the loco-manipulation trajectories on the fly to avoid unexpected interactions with a dynamic environment. This extended abstract provides a pipeline to offline plan a configuration space global trajectory based on a randomized strategy, and to online locally refine it depending on any change of the dynamic environment and the robot state. The offline planner directly plans in the contact space, and additionally seeks for whole-body feasible configurations compliant with the sampled contact states. The planned trajectory, made by a discrete set of contacts and configurations, can be seen as a graph and it can be online refined during the execution of the global trajectory. The online refinement is carried out by a graph optimization planner exploiting visual information. It locally acts on the global initial plan to account for possible changes in the environment. While the offline planner is a concluded work, tested on the humanoid COMAN+, the online local planner is still a work-in-progress which has been tested on a reduced model of the CENTAURO robot to avoid dynamic and static obstacles interfering with a wheeled motion task. Both the COMAN+ and the CENTAURO robots have been designed at the Italian Institute of Technology (IIT).
△ Less
Submitted 20 May, 2022;
originally announced May 2022.
-
Optimization-Based Quadrupedal Hybrid Wheeled-Legged Locomotion
Authors:
Italo Belli,
Matteo Parigi Polverini,
Arturo Laurenzi,
Enrico Mingo Hoffman,
Paolo Rocco,
Nikolaos Tsagarakis
Abstract:
Hybrid wheeled-legged locomotion is a navigation paradigm only recently opened up by novel robotic designs,e.g. the centaur-type humanoid CENTAURO [1] or the quadruped ANYmal [2] in its configuration featuring non-steerable wheels. The term Hybrid Locomotion is hereafter used to indicate a particular type of locomotion, achieved with simultaneous and coordinate use of legs and wheels,see Fig. 1. S…
▽ More
Hybrid wheeled-legged locomotion is a navigation paradigm only recently opened up by novel robotic designs,e.g. the centaur-type humanoid CENTAURO [1] or the quadruped ANYmal [2] in its configuration featuring non-steerable wheels. The term Hybrid Locomotion is hereafter used to indicate a particular type of locomotion, achieved with simultaneous and coordinate use of legs and wheels,see Fig. 1. Such choice stems at the intersection between legged locomotion and the simpler wheeled navigation, in order to get the best from both techniques: agility and ability to traverse uneven terrains from the first, speed and stability from the second. As a consequence, the problem of planning feasible trajectories for a hybrid robot shares many similarities with the legged locomotion problem: also in the hybrid case the motion of the base is reached through contact of the feet with the environment, taking into account that the wheeled feet can just push on the ground and not pull it. Forces compatible with friction cones have to be considered, while the contacts can slide just along the direction prescribed by the orientation of the wheels.
△ Less
Submitted 15 July, 2021;
originally announced July 2021.
-
Agile Actions with a Centaur-Type Humanoid: A Decoupled Approach
Authors:
Matteo Parigi Polverini,
Enrico Mingo Hoffman,
Arturo Laurenzi,
Nikos G. Tsagarakis
Abstract:
The kinematic features of a centaur-type humanoid platform, combined with a powerful actuation, enable the experimentation of a variety of agile and dynamic motions. However, the higher number of degrees-of-freedom and the increased weight of the system, compared to the bipedal and quadrupedal counterparts, pose significant research challenges in terms of computational load and real implementation…
▽ More
The kinematic features of a centaur-type humanoid platform, combined with a powerful actuation, enable the experimentation of a variety of agile and dynamic motions. However, the higher number of degrees-of-freedom and the increased weight of the system, compared to the bipedal and quadrupedal counterparts, pose significant research challenges in terms of computational load and real implementation. To this end, this work presents a control architecture to perform agile actions, conceived for torque-controlled platforms, which decouples for computational purposes offline optimal control planning of lower-body primitives, based on a template kinematic model, and online control of the upper-body motion to maintain balance. Three stabilizing strategies are presented, whose performance is compared in two types of simulated jumps, while experimental validation is performed on a half-squat jump using the CENTAURO robot.
△ Less
Submitted 12 March, 2021;
originally announced March 2021.
-
Curved patch mapping and tracking for irregular terrain modeling: Application to bipedal robot foot placement
Authors:
Dimitrios Kanoulas,
Nikos G. Tsagarakis,
Marsette Vona
Abstract:
Legged robots need to make contact with irregular surfaces, when operating in unstructured natural terrains. Representing and perceiving these areas to reason about potential contact between a robot and its surrounding environment, is still largely an open problem. This paper introduces a new framework to model and map local rough terrain surfaces, for tasks such as bipedal robot foot placement. T…
▽ More
Legged robots need to make contact with irregular surfaces, when operating in unstructured natural terrains. Representing and perceiving these areas to reason about potential contact between a robot and its surrounding environment, is still largely an open problem. This paper introduces a new framework to model and map local rough terrain surfaces, for tasks such as bipedal robot foot placement. The system operates in real-time, on data from an RGB-D and an IMU sensor. We introduce a set of parametrized patch models and an algorithm to fit them in the environment. Potential contacts are identified as bounded curved patches of approximately the same size as the robot's foot sole. This includes sparse seed point sampling, point cloud neighborhood search, and patch fitting and validation. We also present a mapping and tracking system, where patches are maintained in a local spatial map around the robot as it moves. A bio-inspired sampling algorithm is introduced for finding salient contacts. We include a dense volumetric fusion layer for spatiotemporally tracking, using multiple depth data to reconstruct a local point cloud. We present experimental results on a mini-biped robot that performs foot placements on rocks, implementing a 3D foothold perception system, that uses the developed patch mapping and tracking framework.
△ Less
Submitted 5 April, 2020;
originally announced April 2020.
-
Flexible Disaster Response of Tomorrow -- Final Presentation and Evaluation of the CENTAURO System
Authors:
Tobias Klamt,
Diego Rodriguez,
Lorenzo Baccelliere,
Xi Chen,
Domenico Chiaradia,
Torben Cichon,
Massimiliano Gabardi,
Paolo Guria,
Karl Holmquist,
Malgorzata Kamedula,
Hakan Karaoguz,
Navvab Kashiri,
Arturo Laurenzi,
Christian Lenz,
Daniele Leonardis,
Enrico Mingo Hoffman,
Luca Muratore,
Dmytro Pavlichenko,
Francesco Porcini,
Zeyu Ren,
Fabian Schilling,
Max Schwarz,
Massimiliano Solazzi,
Michael Felsberg,
Antonio Frisoli
, et al. (7 additional authors not shown)
Abstract:
Mobile manipulation robots have high potential to support rescue forces in disaster-response missions. Despite the difficulties imposed by real-world scenarios, robots are promising to perform mission tasks from a safe distance. In the CENTAURO project, we developed a disaster-response system which consists of the highly flexible Centauro robot and suitable control interfaces including an immersiv…
▽ More
Mobile manipulation robots have high potential to support rescue forces in disaster-response missions. Despite the difficulties imposed by real-world scenarios, robots are promising to perform mission tasks from a safe distance. In the CENTAURO project, we developed a disaster-response system which consists of the highly flexible Centauro robot and suitable control interfaces including an immersive tele-presence suit and support-operator controls on different levels of autonomy.
In this article, we give an overview of the final CENTAURO system. In particular, we explain several high-level design decisions and how those were derived from requirements and extensive experience of Kerntechnische Hilfsdienst GmbH, Karlsruhe, Germany (KHG). We focus on components which were recently integrated and report about a systematic evaluation which demonstrated system capabilities and revealed valuable insights.
△ Less
Submitted 19 September, 2019;
originally announced September 2019.
-
Remote Mobile Manipulation with the Centauro Robot: Full-body Telepresence and Autonomous Operator Assistance
Authors:
Tobias Klamt,
Max Schwarz,
Christian Lenz,
Lorenzo Baccelliere,
Domenico Buongiorno,
Torben Cichon,
Antonio Di Guardo,
David Droeschel,
Massimiliano Gabardi,
Malgorzata Kamedula,
Navvab Kashiri,
Arturo Laurenzi,
Daniele Leonardis,
Luca Muratore,
Dmytro Pavlichenko,
Arul Selvam Periyasamy,
Diego Rodriguez,
Massimiliano Solazzi,
Antonio Frisoli,
Michael Gustmann,
Jürgen Roßmann,
Uwe Süss,
Nikos G. Tsagarakis,
Sven Behnke
Abstract:
Solving mobile manipulation tasks in inaccessible and dangerous environments is an important application of robots to support humans. Example domains are construction and maintenance of manned and unmanned stations on the moon and other planets. Suitable platforms require flexible and robust hardware, a locomotion approach that allows for navigating a wide variety of terrains, dexterous manipulati…
▽ More
Solving mobile manipulation tasks in inaccessible and dangerous environments is an important application of robots to support humans. Example domains are construction and maintenance of manned and unmanned stations on the moon and other planets. Suitable platforms require flexible and robust hardware, a locomotion approach that allows for navigating a wide variety of terrains, dexterous manipulation capabilities, and respective user interfaces. We present the CENTAURO system which has been designed for these requirements and consists of the Centauro robot and a set of advanced operator interfaces with complementary strength enabling the system to solve a wide range of realistic mobile manipulation tasks. The robot possesses a centaur-like body plan and is driven by torque-controlled compliant actuators. Four articulated legs ending in steerable wheels allow for omnidirectional driving as well as for making steps. An anthropomorphic upper body with two arms ending in five-finger hands enables human-like manipulation. The robot perceives its environment through a suite of multimodal sensors. The resulting platform complexity goes beyond the complexity of most known systems which puts the focus on a suitable operator interface. An operator can control the robot through a telepresence suit, which allows for flexibly solving a large variety of mobile manipulation tasks. Locomotion and manipulation functionalities on different levels of autonomy support the operation. The proposed user interfaces enable solving a wide variety of tasks without previous task-specific training. The integrated system is evaluated in numerous teleoperated experiments that are described along with lessons learned.
△ Less
Submitted 5 August, 2019;
originally announced August 2019.
-
V2CNet: A Deep Learning Framework to Translate Videos to Commands for Robotic Manipulation
Authors:
Anh Nguyen,
Thanh-Toan Do,
Ian Reid,
Darwin G. Caldwell,
Nikos G. Tsagarakis
Abstract:
We propose V2CNet, a new deep learning framework to automatically translate the demonstration videos to commands that can be directly used in robotic applications. Our V2CNet has two branches and aims at understanding the demonstration video in a fine-grained manner. The first branch has the encoder-decoder architecture to encode the visual features and sequentially generate the output words as a…
▽ More
We propose V2CNet, a new deep learning framework to automatically translate the demonstration videos to commands that can be directly used in robotic applications. Our V2CNet has two branches and aims at understanding the demonstration video in a fine-grained manner. The first branch has the encoder-decoder architecture to encode the visual features and sequentially generate the output words as a command, while the second branch uses a Temporal Convolutional Network (TCN) to learn the fine-grained actions. By jointly training both branches, the network is able to model the sequential information of the command, while effectively encodes the fine-grained actions. The experimental results on our new large-scale dataset show that V2CNet outperforms recent state-of-the-art methods by a substantial margin, while its output can be applied in real robotic applications. The source code and trained models will be made available.
△ Less
Submitted 23 March, 2019;
originally announced March 2019.
-
Nonlinear Model Predictive Control for Robust Bipedal Locomotion: Exploring Angular Momentum and CoM Height Changes
Authors:
Jiatao Ding,
Chengxu Zhou,
Songyan Xin,
Xiaohui Xiao,
Nikos Tsagarakis
Abstract:
Human beings can utilize multiple balance strategies, e.g. step location adjustment and angular momentum adaptation, to maintain balance when walking under dynamic disturbances. In this work, we propose a novel Nonlinear Model Predictive Control (NMPC) framework for robust locomotion, with the capabilities of step location adjustment, Center of Mass (CoM) height variation, and angular momentum ada…
▽ More
Human beings can utilize multiple balance strategies, e.g. step location adjustment and angular momentum adaptation, to maintain balance when walking under dynamic disturbances. In this work, we propose a novel Nonlinear Model Predictive Control (NMPC) framework for robust locomotion, with the capabilities of step location adjustment, Center of Mass (CoM) height variation, and angular momentum adaptation. These features are realized by constraining the Zero Moment Point within the support polygon. By using the nonlinear inverted pendulum plus flywheel model, the effects of upper-body rotation and vertical height motion are considered. As a result, the NMPC is formulated as a quadratically constrained quadratic program problem, which is solved fast by sequential quadratic programming. Using this unified framework, robust walking patterns that exploit reactive stepping, body inclination, and CoM height variation are generated based on the state estimation. The adaptability for bipedal walking in multiple scenarios has been demonstrated through simulation studies.
△ Less
Submitted 24 January, 2021; v1 submitted 18 February, 2019;
originally announced February 2019.
-
Object Captioning and Retrieval with Natural Language
Authors:
Anh Nguyen,
Thanh-Toan Do,
Ian Reid,
Darwin G. Caldwell,
Nikos G. Tsagarakis
Abstract:
We address the problem of jointly learning vision and language to understand the object in a fine-grained manner. The key idea of our approach is the use of object descriptions to provide the detailed understanding of an object. Based on this idea, we propose two new architectures to solve two related problems: object captioning and natural language-based object retrieval. The goal of the object c…
▽ More
We address the problem of jointly learning vision and language to understand the object in a fine-grained manner. The key idea of our approach is the use of object descriptions to provide the detailed understanding of an object. Based on this idea, we propose two new architectures to solve two related problems: object captioning and natural language-based object retrieval. The goal of the object captioning task is to simultaneously detect the object and generate its associated description, while in the object retrieval task, the goal is to localize an object given an input query. We demonstrate that both problems can be solved effectively using hybrid end-to-end CNN-LSTM networks. The experimental results on our new challenging dataset show that our methods outperform recent methods by a fair margin, while providing a detailed understanding of the object and having fast inference time. The source code will be made available.
△ Less
Submitted 16 March, 2018;
originally announced March 2018.
-
Center-of-Mass-Based Grasp Pose Adaptation Using 3D Range and Force/Torque Sensing
Authors:
Dimitrios Kanoulas,
Jinoh Lee,
Darwin G. Caldwell,
Nikos G. Tsagarakis
Abstract:
Lifting objects, whose mass may produce high wrist torques that exceed the hardware strength limits, could lead to unstable grasps or serious robot damage. This work introduces a new Center-of-Mass (CoM)-based grasp pose adaptation method, for picking up objects using a combination of exteroceptive 3D perception and proprioceptive force/torque sensor feedback. The method works in two iterative sta…
▽ More
Lifting objects, whose mass may produce high wrist torques that exceed the hardware strength limits, could lead to unstable grasps or serious robot damage. This work introduces a new Center-of-Mass (CoM)-based grasp pose adaptation method, for picking up objects using a combination of exteroceptive 3D perception and proprioceptive force/torque sensor feedback. The method works in two iterative stages to provide reliable and wrist torque efficient grasps. Initially, a geometric object CoM is estimated from the input range data. In the first stage, a set of hand-size handle grasps are localized on the object and the closest to its CoM is selected for grasping. In the second stage, the object is lifted using a single arm, while the force and torque readings from the sensor on the wrist are monitored. Based on these readings, a displacement to the new CoM estimation is calculated. The object is released and the process is repeated until the wrist torque effort is minimized. The advantage of our method is the blending of both exteroceptive (3D range) and proprioceptive (force/torque) sensing for finding the grasp location that minimizes the wrist effort, potentially improving the reliability of the grasping and the subsequent manipulation task. We experimentally validate the proposed method by executing a number of tests on a set of objects that include handles, using the humanoid robot WALK-MAN.
△ Less
Submitted 18 February, 2018;
originally announced February 2018.
-
Artificial Delay Based ARC of a Class of Uncertain EL Systems with Only Position Feedback
Authors:
Spandan Roy,
Indra Narayan Kar,
Jinoh Lee,
Nikos Tsagarakis,
Darwin G. Caldwell
Abstract:
In this paper, the tracking control problem of an Euler-Lagrange system is addressed with regard to parametric uncertainties, and an adaptive-robust control strategy, christened Time-Delayed Adaptive Robust Control (TARC), is presented. TARC approximates the unknown dynamics through the time-delayed estimation, and the adaptive-robust control provides robustness against the approximation error. Th…
▽ More
In this paper, the tracking control problem of an Euler-Lagrange system is addressed with regard to parametric uncertainties, and an adaptive-robust control strategy, christened Time-Delayed Adaptive Robust Control (TARC), is presented. TARC approximates the unknown dynamics through the time-delayed estimation, and the adaptive-robust control provides robustness against the approximation error. The novel adaptation law of TARC, in contrast to the conventional adaptive-robust control methodologies, requires neither complete model of the system nor any knowledge of predefined uncertainty bounds to compute the switching gain, and circumvents the over- and underestimation problems of the switching gain. Moreover, TARC only utilizes position feedback and approximates the velocity and acceleration terms from the past position data. The adopted state-derivatives estimation method in TARC avoids any explicit requirement of external low pass filters for the removal of measurement noise. A new stability notion in continuous-time domain is proposed considering the time delay, adaptive law, and state-derivatives estimation which in turn provides a selection criterion for gains and sampling interval of the controller.
△ Less
Submitted 3 November, 2017;
originally announced November 2017.
-
Translating Videos to Commands for Robotic Manipulation with Deep Recurrent Neural Networks
Authors:
Anh Nguyen,
Dimitrios Kanoulas,
Luca Muratore,
Darwin G. Caldwell,
Nikos G. Tsagarakis
Abstract:
We present a new method to translate videos to commands for robotic manipulation using Deep Recurrent Neural Networks (RNN). Our framework first extracts deep features from the input video frames with a deep Convolutional Neural Networks (CNN). Two RNN layers with an encoder-decoder architecture are then used to encode the visual features and sequentially generate the output words as the command.…
▽ More
We present a new method to translate videos to commands for robotic manipulation using Deep Recurrent Neural Networks (RNN). Our framework first extracts deep features from the input video frames with a deep Convolutional Neural Networks (CNN). Two RNN layers with an encoder-decoder architecture are then used to encode the visual features and sequentially generate the output words as the command. We demonstrate that the translation accuracy can be improved by allowing a smooth transaction between two RNN layers and using the state-of-the-art feature extractor. The experimental results on our new challenging dataset show that our approach outperforms recent methods by a fair margin. Furthermore, we combine the proposed translation module with the vision and planning system to let a robot perform various manipulation tasks. Finally, we demonstrate the effectiveness of our framework on a full-size humanoid robot WALK-MAN.
△ Less
Submitted 1 October, 2017;
originally announced October 2017.
-
Real-Time 6DOF Pose Relocalization for Event Cameras with Stacked Spatial LSTM Networks
Authors:
Anh Nguyen,
Thanh-Toan Do,
Darwin G. Caldwell,
Nikos G. Tsagarakis
Abstract:
We present a new method to relocalize the 6DOF pose of an event camera solely based on the event stream. Our method first creates the event image from a list of events that occurs in a very short time interval, then a Stacked Spatial LSTM Network (SP-LSTM) is used to learn the camera pose. Our SP-LSTM is composed of a CNN to learn deep features from the event images and a stack of LSTM to learn sp…
▽ More
We present a new method to relocalize the 6DOF pose of an event camera solely based on the event stream. Our method first creates the event image from a list of events that occurs in a very short time interval, then a Stacked Spatial LSTM Network (SP-LSTM) is used to learn the camera pose. Our SP-LSTM is composed of a CNN to learn deep features from the event images and a stack of LSTM to learn spatial dependencies in the image feature space. We show that the spatial dependency plays an important role in the relocalization task and the SP-LSTM can effectively learn this information. The experimental results on a publicly available dataset show that our approach generalizes well and outperforms recent methods by a substantial margin. Overall, our proposed method reduces by approx. 6 times the position error and 3 times the orientation error compared to the current state of the art. The source code and trained models will be released.
△ Less
Submitted 13 April, 2018; v1 submitted 22 August, 2017;
originally announced August 2017.
