-
Self-centering 3-DOF feet controller for hands-free locomotion control in telepresence and virtual reality
Authors:
Raphael Memmesheimer,
Christian Lenz,
Max Schwarz,
Michael Schreiber,
Sven Behnke
Abstract:
We present a novel seated foot controller for handling 3-DOF aimed to control locomotion for telepresence robotics and virtual reality environments. Tilting the feet on two axes yields in forward, backward and sideways motion. In addition, a separate rotary joint allows for rotation around the vertical axis. Attached springs on all joints self-center the controller. The HTC Vive tracker is used to…
▽ More
We present a novel seated foot controller for handling 3-DOF aimed to control locomotion for telepresence robotics and virtual reality environments. Tilting the feet on two axes yields in forward, backward and sideways motion. In addition, a separate rotary joint allows for rotation around the vertical axis. Attached springs on all joints self-center the controller. The HTC Vive tracker is used to translate the trackers' orientation into locomotion commands. The proposed self-centering foot controller was used successfully for the ANA Avatar XPRIZE competition, where a naive operator traversed the robot through a longer distance, surpassing obstacles while solving various interaction and manipulation tasks in between. We publicly provide the models of the mostly 3D-printed feet controller for reproduction.
△ Less
Submitted 5 August, 2024;
originally announced August 2024.
-
HortiBot: An Adaptive Multi-Arm System for Robotic Horticulture of Sweet Peppers
Authors:
Christian Lenz,
Rohit Menon,
Michael Schreiber,
Melvin Paul Jacob,
Sven Behnke,
Maren Bennewitz
Abstract:
Horticultural tasks such as pruning and selective harvesting are labor intensive and horticultural staff are hard to find. Automating these tasks is challenging due to the semi-structured greenhouse workspaces, changing environmental conditions such as lighting, dense plant growth with many occlusions, and the need for gentle manipulation of non-rigid plant organs. In this work, we present the thr…
▽ More
Horticultural tasks such as pruning and selective harvesting are labor intensive and horticultural staff are hard to find. Automating these tasks is challenging due to the semi-structured greenhouse workspaces, changing environmental conditions such as lighting, dense plant growth with many occlusions, and the need for gentle manipulation of non-rigid plant organs. In this work, we present the three-armed system HortiBot, with two arms for manipulation and a third arm as an articulated head for active perception using stereo cameras. Its perception system detects not only peppers, but also peduncles and stems in real time, and performs online data association to build a world model of pepper plants. Collision-aware online trajectory generation allows all three arms to safely track their respective targets for observation, grasping, and cutting. We integrated perception and manipulation to perform selective harvesting of peppers and evaluated the system in lab experiments. Using active perception coupled with end-effector force torque sensing for compliant manipulation, HortiBot achieves high success rates.
△ Less
Submitted 22 March, 2024;
originally announced March 2024.
-
NimbRo wins ANA Avatar XPRIZE Immersive Telepresence Competition: Human-Centric Evaluation and Lessons Learned
Authors:
Christian Lenz,
Max Schwarz,
Andre Rochow,
Bastian Pätzold,
Raphael Memmesheimer,
Michael Schreiber,
Sven Behnke
Abstract:
Robotic avatar systems can enable immersive telepresence with locomotion, manipulation, and communication capabilities. We present such an avatar system, based on the key components of immersive 3D visualization and transparent force-feedback telemanipulation. Our avatar robot features an anthropomorphic upper body with dexterous hands. The remote human operator drives the arms and fingers through…
▽ More
Robotic avatar systems can enable immersive telepresence with locomotion, manipulation, and communication capabilities. We present such an avatar system, based on the key components of immersive 3D visualization and transparent force-feedback telemanipulation. Our avatar robot features an anthropomorphic upper body with dexterous hands. The remote human operator drives the arms and fingers through an exoskeleton-based operator station, which provides force feedback both at the wrist and for each finger. The robot torso is mounted on a holonomic base, providing omnidirectional locomotion on flat floors, controlled using a 3D rudder device. Finally, the robot features a 6D movable head with stereo cameras, which stream images to a VR display worn by the operator. Movement latency is hidden using spherical rendering. The head also carries a telepresence screen displaying an animated image of the operator's face, enabling direct interaction with remote persons. Our system won the \$10M ANA Avatar XPRIZE competition, which challenged teams to develop intuitive and immersive avatar systems that could be operated by briefly trained judges. We analyze our successful participation in the semifinals and finals and provide insight into our operator training and lessons learned. In addition, we evaluate our system in a user study that demonstrates its intuitive and easy usability.
△ Less
Submitted 28 August, 2023; v1 submitted 23 August, 2023;
originally announced August 2023.
-
Audio-based Roughness Sensing and Tactile Feedback for Haptic Perception in Telepresence
Authors:
Bastian Pätzold,
Andre Rochow,
Michael Schreiber,
Raphael Memmesheimer,
Christian Lenz,
Max Schwarz,
Sven Behnke
Abstract:
Haptic perception is highly important for immersive teleoperation of robots, especially for accomplishing manipulation tasks. We propose a low-cost haptic sensing and rendering system, which is capable of detecting and displaying surface roughness. As the robot fingertip moves across a surface of interest, two microphones capture sound coupled directly through the fingertip and through the air, re…
▽ More
Haptic perception is highly important for immersive teleoperation of robots, especially for accomplishing manipulation tasks. We propose a low-cost haptic sensing and rendering system, which is capable of detecting and displaying surface roughness. As the robot fingertip moves across a surface of interest, two microphones capture sound coupled directly through the fingertip and through the air, respectively. A learning-based detector system analyzes the data in real time and gives roughness estimates with both high temporal resolution and low latency. Finally, an audio-based vibrational actuator displays the result to the human operator. We demonstrate the effectiveness of our system through lab experiments and our winning entry in the ANA Avatar XPRIZE competition finals, where briefly trained judges solved a roughness-based selection task even without additional vision feedback. We publish our dataset used for training and evaluation together with our trained models to enable reproducibility of results.
△ Less
Submitted 16 October, 2023; v1 submitted 13 March, 2023;
originally announced March 2023.
-
Robust Immersive Telepresence and Mobile Telemanipulation: NimbRo wins ANA Avatar XPRIZE Finals
Authors:
Max Schwarz,
Christian Lenz,
Raphael Memmesheimer,
Bastian Pätzold,
Andre Rochow,
Michael Schreiber,
Sven Behnke
Abstract:
Robotic avatar systems promise to bridge distances and reduce the need for travel. We present the updated NimbRo avatar system, winner of the $5M grand prize at the international ANA Avatar XPRIZE competition, which required participants to build intuitive and immersive robotic telepresence systems that could be operated by briefly trained operators. We describe key improvements for the finals, co…
▽ More
Robotic avatar systems promise to bridge distances and reduce the need for travel. We present the updated NimbRo avatar system, winner of the $5M grand prize at the international ANA Avatar XPRIZE competition, which required participants to build intuitive and immersive robotic telepresence systems that could be operated by briefly trained operators. We describe key improvements for the finals, compared to the system used in the semifinals: To operate without a power- and communications tether, we integrated a battery and a robust redundant wireless communication system. Video and audio data are compressed using low-latency HEVC and Opus codecs. We propose a new locomotion control device with tunable resistance force. To increase flexibility, the robot's upper-body height can be adjusted by the operator. We describe essential monitoring and robustness tools which enabled the success at the competition. Finally, we analyze our performance at the competition finals and discuss lessons learned.
△ Less
Submitted 6 December, 2023; v1 submitted 6 March, 2023;
originally announced March 2023.
-
Bimanual Telemanipulation with Force and Haptic Feedback through an Anthropomorphic Avatar System
Authors:
Christian Lenz,
Sven Behnke
Abstract:
Robotic teleoperation is a key technology for a wide variety of applications. It allows sending robots instead of humans in remote, possibly dangerous locations while still using the human brain with its enormous knowledge and creativity, especially for solving unexpected problems. A main challenge in teleoperation consists of providing enough feedback to the human operator for situation awareness…
▽ More
Robotic teleoperation is a key technology for a wide variety of applications. It allows sending robots instead of humans in remote, possibly dangerous locations while still using the human brain with its enormous knowledge and creativity, especially for solving unexpected problems. A main challenge in teleoperation consists of providing enough feedback to the human operator for situation awareness and thus create full immersion, as well as offering the operator suitable control interfaces to achieve efficient and robust task fulfillment. We present a bimanual telemanipulation system consisting of an anthropomorphic avatar robot and an operator station providing force and haptic feedback to the human operator. The avatar arms are controlled in Cartesian space with a direct mapping of the operator movements. The measured forces and torques on the avatar side are haptically displayed to the operator. We developed a predictive avatar model for limit avoidance which runs on the operator side, ensuring low latency. The system was successfully evaluated during the ANA Avatar XPRIZE competition semifinals. In addition, we performed in lab experiments and carried out a small user study with mostly untrained operators.
△ Less
Submitted 2 January, 2023;
originally announced January 2023.
-
NimbRo Avatar: Interactive Immersive Telepresence with Force-Feedback Telemanipulation
Authors:
Max Schwarz,
Christian Lenz,
Andre Rochow,
Michael Schreiber,
Sven Behnke
Abstract:
Robotic avatars promise immersive teleoperation with human-like manipulation and communication capabilities. We present such an avatar system, based on the key components of immersive 3D visualization and transparent force-feedback telemanipulation. Our avatar robot features an anthropomorphic bimanual arm configuration with dexterous hands. The remote human operator drives the arms and fingers th…
▽ More
Robotic avatars promise immersive teleoperation with human-like manipulation and communication capabilities. We present such an avatar system, based on the key components of immersive 3D visualization and transparent force-feedback telemanipulation. Our avatar robot features an anthropomorphic bimanual arm configuration with dexterous hands. The remote human operator drives the arms and fingers through an exoskeleton-based operator station, which provides force feedback both at the wrist and for each finger. The robot torso is mounted on a holonomic base, providing locomotion capability in typical indoor scenarios, controlled using a 3D rudder device. Finally, the robot features a 6D movable head with stereo cameras, which stream images to a VR HMD worn by the operator. Movement latency is hidden using spherical rendering. The head also carries a telepresence screen displaying a synthesized image of the operator with facial animation, which enables direct interaction with remote persons. We evaluate our system successfully both in a user study with untrained operators as well as a longer and more complex integrated mission. We discuss lessons learned from the trials and possible improvements.
△ Less
Submitted 28 September, 2021;
originally announced September 2021.
-
Bimanual Telemanipulation with Force and Haptic Feedback and Predictive Limit Avoidance
Authors:
Christian Lenz,
Sven Behnke
Abstract:
Robotic teleoperation is a key technology for a wide variety of applications. It allows sending robots instead of humans in remote, possibly dangerous locations while still using the human brain with its enormous knowledge and creativity, especially for solving unexpected problems. A main challenge in teleoperation consists of providing enough feedback to the human operator for situation awareness…
▽ More
Robotic teleoperation is a key technology for a wide variety of applications. It allows sending robots instead of humans in remote, possibly dangerous locations while still using the human brain with its enormous knowledge and creativity, especially for solving unexpected problems. A main challenge in teleoperation consists of providing enough feedback to the human operator for situation awareness and thus create full immersion, as well as offering the operator suitable control interfaces to achieve efficient and robust task fulfillment. We present a bimanual telemanipulation system consisting of an anthropomorphic avatar robot and an operator station providing force and haptic feedback to the human operator. The avatar arms are controlled in Cartesian space with a 1:1 mapping of the operator movements. The measured forces and torques on the avatar side are haptically displayed directly to the operator. We developed a predictive avatar model for limit avoidance which runs on the operator side, ensuring low latency. Only off-the-shelf components were used to build the system. It is evaluated in lab experiments and by untrained operators in a small user study.
△ Less
Submitted 27 September, 2021;
originally announced September 2021.
-
Autonomous Fire Fighting with a UAV-UGV Team at MBZIRC 2020
Authors:
Jan Quenzel,
Malte Splietker,
Dmytro Pavlichenko,
Daniel Schleich,
Christian Lenz,
Max Schwarz,
Michael Schreiber,
Marius Beul,
Sven Behnke
Abstract:
Every day, burning buildings threaten the lives of occupants and first responders trying to save them. Quick action is of essence, but some areas might not be accessible or too dangerous to enter. Robotic systems have become a promising addition to firefighting, but at this stage, they are mostly manually controlled, which is error-prone and requires specially trained personal.
We present two sy…
▽ More
Every day, burning buildings threaten the lives of occupants and first responders trying to save them. Quick action is of essence, but some areas might not be accessible or too dangerous to enter. Robotic systems have become a promising addition to firefighting, but at this stage, they are mostly manually controlled, which is error-prone and requires specially trained personal.
We present two systems for autonomous firefighting from air and ground we developed for the Mohamed Bin Zayed International Robotics Challenge (MBZIRC) 2020. The systems use LiDAR for reliable localization within narrow, potentially GNSS-restricted environments while maneuvering close to obstacles. Measurements from LiDAR and thermal cameras are fused to track fires, while relative navigation ensures successful extinguishing.
We analyze and discuss our successful participation during the MBZIRC 2020, present further experiments, and provide insights into our lessons learned from the competition.
△ Less
Submitted 11 June, 2021;
originally announced June 2021.
-
Team NimbRo's UGV Solution for Autonomous Wall Building and Fire Fighting at MBZIRC 2020
Authors:
Christian Lenz,
Jan Quenzel,
Arul Selvam Periyasamy,
Jan Razlaw,
Andre Rochow,
Malte Splietker,
Michael Schreiber,
Max Schwarz,
Finn Süberkrüb,
Sven Behnke
Abstract:
Autonomous robotic systems for various applications including transport, mobile manipulation, and disaster response are becoming more and more complex. Evaluating and analyzing such systems is challenging. Robotic competitions are designed to benchmark complete robotic systems on complex state-of-the-art tasks. Participants compete in defined scenarios under equal conditions. We present our UGV so…
▽ More
Autonomous robotic systems for various applications including transport, mobile manipulation, and disaster response are becoming more and more complex. Evaluating and analyzing such systems is challenging. Robotic competitions are designed to benchmark complete robotic systems on complex state-of-the-art tasks. Participants compete in defined scenarios under equal conditions. We present our UGV solution developed for the Mohamed Bin Zayed International Robotics Challenge 2020. Our hard- and software components to address the challenge tasks of wall building and fire fighting are integrated into a fully autonomous system. The robot consists of a wheeled omnidirectional base, a 6 DoF manipulator arm equipped with a magnetic gripper, a highly efficient storage system to transport box-shaped objects, and a water spraying system to fight fires. The robot perceives its environment using 3D LiDAR as well as RGB and thermal camera-based perception modules, is capable of picking box-shaped objects and constructing a pre-defined wall structure, as well as detecting and localizing heat sources in order to extinguish potential fires. A high-level planner solves the challenge tasks using the robot skills. We analyze and discuss our successful participation during the MBZIRC 2020 finals, present further experiments, and provide insights to our lessons learned.
△ Less
Submitted 27 May, 2021; v1 submitted 25 May, 2021;
originally announced May 2021.
-
Autonomous Wall Building with a UGV-UAV Team at MBZIRC 2020
Authors:
Christian Lenz,
Max Schwarz,
Andre Rochow,
Jan Razlaw,
Arul Selvam Periyasamy,
Michael Schreiber,
Sven Behnke
Abstract:
Constructing large structures with robots is a challenging task with many potential applications that requires mobile manipulation capabilities. We present two systems for autonomous wall building that we developed for the Mohamed Bin Zayed International Robotics Challenge 2020. Both systems autonomously perceive their environment, find bricks, and build a predefined wall structure. While the UGV…
▽ More
Constructing large structures with robots is a challenging task with many potential applications that requires mobile manipulation capabilities. We present two systems for autonomous wall building that we developed for the Mohamed Bin Zayed International Robotics Challenge 2020. Both systems autonomously perceive their environment, find bricks, and build a predefined wall structure. While the UGV uses a 3D LiDAR-based perception system which measures brick poses with high precision, the UAV employs a real-time camera-based system for visual servoing. We report results and insights from our successful participation at the MBZIRC 2020 Finals, additional lab experiments, and discuss the lessons learned from the competition.
△ Less
Submitted 3 November, 2020;
originally announced November 2020.
-
Autonomous Bimanual Functional Regrasping of Novel Object Class Instances
Authors:
Dmytro Pavlichenko,
Diego Rodriguez,
Christian Lenz,
Max Schwarz,
Sven Behnke
Abstract:
In human-made scenarios, robots need to be able to fully operate objects in their surroundings, i.e., objects are required to be functionally grasped rather than only picked. This imposes very strict constraints on the object pose such that a direct grasp can be performed. Inspired by the anthropomorphic nature of humanoid robots, we propose an approach that first grasps an object with one hand, o…
▽ More
In human-made scenarios, robots need to be able to fully operate objects in their surroundings, i.e., objects are required to be functionally grasped rather than only picked. This imposes very strict constraints on the object pose such that a direct grasp can be performed. Inspired by the anthropomorphic nature of humanoid robots, we propose an approach that first grasps an object with one hand, obtaining full control over its pose, and performs the functional grasp with the second hand subsequently. Thus, we develop a fully autonomous pipeline for dual-arm functional regrasping of novel familiar objects, i.e., objects never seen before that belong to a known object category, e.g., spray bottles. This process involves semantic segmentation, object pose estimation, non-rigid mesh registration, grasp sampling, handover pose generation and in-hand pose refinement. The latter is used to compensate for the unpredictable object movement during the first grasp. The approach is applied to a human-like upper body. To the best knowledge of the authors, this is the first system that exhibits autonomous bimanual functional regrasping capabilities. We demonstrate that our system yields reliable success rates and can be applied on-line to real-world tasks using only one off-the-shelf RGB-D sensor.
△ Less
Submitted 1 October, 2019;
originally announced October 2019.
-
Flexible Disaster Response of Tomorrow -- Final Presentation and Evaluation of the CENTAURO System
Authors:
Tobias Klamt,
Diego Rodriguez,
Lorenzo Baccelliere,
Xi Chen,
Domenico Chiaradia,
Torben Cichon,
Massimiliano Gabardi,
Paolo Guria,
Karl Holmquist,
Malgorzata Kamedula,
Hakan Karaoguz,
Navvab Kashiri,
Arturo Laurenzi,
Christian Lenz,
Daniele Leonardis,
Enrico Mingo Hoffman,
Luca Muratore,
Dmytro Pavlichenko,
Francesco Porcini,
Zeyu Ren,
Fabian Schilling,
Max Schwarz,
Massimiliano Solazzi,
Michael Felsberg,
Antonio Frisoli
, et al. (7 additional authors not shown)
Abstract:
Mobile manipulation robots have high potential to support rescue forces in disaster-response missions. Despite the difficulties imposed by real-world scenarios, robots are promising to perform mission tasks from a safe distance. In the CENTAURO project, we developed a disaster-response system which consists of the highly flexible Centauro robot and suitable control interfaces including an immersiv…
▽ More
Mobile manipulation robots have high potential to support rescue forces in disaster-response missions. Despite the difficulties imposed by real-world scenarios, robots are promising to perform mission tasks from a safe distance. In the CENTAURO project, we developed a disaster-response system which consists of the highly flexible Centauro robot and suitable control interfaces including an immersive tele-presence suit and support-operator controls on different levels of autonomy.
In this article, we give an overview of the final CENTAURO system. In particular, we explain several high-level design decisions and how those were derived from requirements and extensive experience of Kerntechnische Hilfsdienst GmbH, Karlsruhe, Germany (KHG). We focus on components which were recently integrated and report about a systematic evaluation which demonstrated system capabilities and revealed valuable insights.
△ Less
Submitted 19 September, 2019;
originally announced September 2019.
-
A VR System for Immersive Teleoperation and Live Exploration with a Mobile Robot
Authors:
Patrick Stotko,
Stefan Krumpen,
Max Schwarz,
Christian Lenz,
Sven Behnke,
Reinhard Klein,
Michael Weinmann
Abstract:
Applications like disaster management and industrial inspection often require experts to enter contaminated places. To circumvent the need for physical presence, it is desirable to generate a fully immersive individual live teleoperation experience. However, standard video-based approaches suffer from a limited degree of immersion and situation awareness due to the restriction to the camera view,…
▽ More
Applications like disaster management and industrial inspection often require experts to enter contaminated places. To circumvent the need for physical presence, it is desirable to generate a fully immersive individual live teleoperation experience. However, standard video-based approaches suffer from a limited degree of immersion and situation awareness due to the restriction to the camera view, which impacts the navigation. In this paper, we present a novel VR-based practical system for immersive robot teleoperation and scene exploration. While being operated through the scene, a robot captures RGB-D data that is streamed to a SLAM-based live multi-client telepresence system. Here, a global 3D model of the already captured scene parts is reconstructed and streamed to the individual remote user clients where the rendering for e.g. head-mounted display devices (HMDs) is performed. We introduce a novel lightweight robot client component which transmits robot-specific data and enables a quick integration into existing robotic systems. This way, in contrast to first-person exploration systems, the operators can explore and navigate in the remote site completely independent of the current position and view of the capturing robot, complementing traditional input devices for teleoperation. We provide a proof-of-concept implementation and demonstrate the capabilities as well as the performance of our system regarding interactive object measurements and bandwidth-efficient data streaming and visualization. Furthermore, we show its benefits over purely video-based teleoperation in a user study revealing a higher degree of situation awareness and a more precise navigation in challenging environments.
△ Less
Submitted 3 February, 2020; v1 submitted 8 August, 2019;
originally announced August 2019.
-
Remote Mobile Manipulation with the Centauro Robot: Full-body Telepresence and Autonomous Operator Assistance
Authors:
Tobias Klamt,
Max Schwarz,
Christian Lenz,
Lorenzo Baccelliere,
Domenico Buongiorno,
Torben Cichon,
Antonio Di Guardo,
David Droeschel,
Massimiliano Gabardi,
Malgorzata Kamedula,
Navvab Kashiri,
Arturo Laurenzi,
Daniele Leonardis,
Luca Muratore,
Dmytro Pavlichenko,
Arul Selvam Periyasamy,
Diego Rodriguez,
Massimiliano Solazzi,
Antonio Frisoli,
Michael Gustmann,
Jürgen Roßmann,
Uwe Süss,
Nikos G. Tsagarakis,
Sven Behnke
Abstract:
Solving mobile manipulation tasks in inaccessible and dangerous environments is an important application of robots to support humans. Example domains are construction and maintenance of manned and unmanned stations on the moon and other planets. Suitable platforms require flexible and robust hardware, a locomotion approach that allows for navigating a wide variety of terrains, dexterous manipulati…
▽ More
Solving mobile manipulation tasks in inaccessible and dangerous environments is an important application of robots to support humans. Example domains are construction and maintenance of manned and unmanned stations on the moon and other planets. Suitable platforms require flexible and robust hardware, a locomotion approach that allows for navigating a wide variety of terrains, dexterous manipulation capabilities, and respective user interfaces. We present the CENTAURO system which has been designed for these requirements and consists of the Centauro robot and a set of advanced operator interfaces with complementary strength enabling the system to solve a wide range of realistic mobile manipulation tasks. The robot possesses a centaur-like body plan and is driven by torque-controlled compliant actuators. Four articulated legs ending in steerable wheels allow for omnidirectional driving as well as for making steps. An anthropomorphic upper body with two arms ending in five-finger hands enables human-like manipulation. The robot perceives its environment through a suite of multimodal sensors. The resulting platform complexity goes beyond the complexity of most known systems which puts the focus on a suitable operator interface. An operator can control the robot through a telepresence suit, which allows for flexibly solving a large variety of mobile manipulation tasks. Locomotion and manipulation functionalities on different levels of autonomy support the operation. The proposed user interfaces enable solving a wide variety of tasks without previous task-specific training. The integrated system is evaluated in numerous teleoperated experiments that are described along with lessons learned.
△ Less
Submitted 5 August, 2019;
originally announced August 2019.
-
Providentia -- A Large-Scale Sensor System for the Assistance of Autonomous Vehicles and Its Evaluation
Authors:
Annkathrin Krämmer,
Christoph Schöller,
Dhiraj Gulati,
Venkatnarayanan Lakshminarasimhan,
Franz Kurz,
Dominik Rosenbaum,
Claus Lenz,
Alois Knoll
Abstract:
The environmental perception of an autonomous vehicle is limited by its physical sensor ranges and algorithmic performance, as well as by occlusions that degrade its understanding of an ongoing traffic situation. This not only poses a significant threat to safety and limits driving speeds, but it can also lead to inconvenient maneuvers. Intelligent Infrastructure Systems can help to alleviate thes…
▽ More
The environmental perception of an autonomous vehicle is limited by its physical sensor ranges and algorithmic performance, as well as by occlusions that degrade its understanding of an ongoing traffic situation. This not only poses a significant threat to safety and limits driving speeds, but it can also lead to inconvenient maneuvers. Intelligent Infrastructure Systems can help to alleviate these problems. An Intelligent Infrastructure System can fill in the gaps in a vehicle's perception and extend its field of view by providing additional detailed information about its surroundings, in the form of a digital model of the current traffic situation, i.e. a digital twin. However, detailed descriptions of such systems and working prototypes demonstrating their feasibility are scarce. In this paper, we propose a hardware and software architecture that enables such a reliable Intelligent Infrastructure System to be built. We have implemented this system in the real world and demonstrate its ability to create an accurate digital twin of an extended highway stretch, thus enhancing an autonomous vehicle's perception beyond the limits of its on-board sensors. Furthermore, we evaluate the accuracy and reliability of the digital twin by using aerial images and earth observation methods for generating ground truth data.
△ Less
Submitted 8 December, 2021; v1 submitted 16 June, 2019;
originally announced June 2019.
-
Autonomous Dual-Arm Manipulation of Familiar Objects
Authors:
Dmytro Pavlichenko,
Diego Rodriguez,
Max Schwarz,
Christian Lenz,
Arul Selvam Periyasamy,
Sven Behnke
Abstract:
Autonomous dual-arm manipulation is an essential skill to deploy robots in unstructured scenarios. However, this is a challenging undertaking, particularly in terms of perception and planning. Unstructured scenarios are full of objects with different shapes and appearances that have to be grasped in a very specific manner so they can be functionally used. In this paper we present an integrated app…
▽ More
Autonomous dual-arm manipulation is an essential skill to deploy robots in unstructured scenarios. However, this is a challenging undertaking, particularly in terms of perception and planning. Unstructured scenarios are full of objects with different shapes and appearances that have to be grasped in a very specific manner so they can be functionally used. In this paper we present an integrated approach to perform dual-arm pick tasks autonomously. Our method consists of semantic segmentation, object pose estimation, deformable model registration, grasp planning and arm trajectory optimization. The entire pipeline can be executed on-board and is suitable for on-line grasping scenarios. For this, our approach makes use of accumulated knowledge expressed as convolutional neural network models and low-dimensional latent shape spaces. For manipulating objects, we propose a stochastic trajectory optimization that includes a kinematic chain closure constraint. Evaluation in simulation and on the real robot corroborates the feasibility and applicability of the proposed methods on a task of picking up unknown watering cans and drills using both arms.
△ Less
Submitted 21 November, 2018;
originally announced November 2018.
-
Team NimbRo at MBZIRC 2017: Autonomous Valve Stem Turning using a Wrench
Authors:
Max Schwarz,
David Droeschel,
Christian Lenz,
Arul Selvam Periyasamy,
En Yen Puang,
Jan Razlaw,
Diego Rodriguez,
Sebastian Schüller,
Michael Schreiber,
Sven Behnke
Abstract:
The Mohamed Bin Zayed International Robotics Challenge (MBZIRC) 2017 has defined ambitious new benchmarks to advance the state-of-the-art in autonomous operation of ground-based and flying robots. In this article, we describe our winning entry to MBZIRC Challenge 2: the mobile manipulation robot Mario. It is capable of autonomously solving a valve manipulation task using a wrench tool detected, gr…
▽ More
The Mohamed Bin Zayed International Robotics Challenge (MBZIRC) 2017 has defined ambitious new benchmarks to advance the state-of-the-art in autonomous operation of ground-based and flying robots. In this article, we describe our winning entry to MBZIRC Challenge 2: the mobile manipulation robot Mario. It is capable of autonomously solving a valve manipulation task using a wrench tool detected, grasped, and finally employed to turn a valve stem. Mario's omnidirectional base allows both fast locomotion and precise close approach to the manipulation panel. We describe an efficient detector for medium-sized objects in 3D laser scans and apply it to detect the manipulation panel. An object detection architecture based on deep neural networks is used to find and select the correct tool from grayscale images. Parametrized motion primitives are adapted online to percepts of the tool and valve stem in order to turn the stem. We report in detail on our winning performance at the challenge and discuss lessons learned.
△ Less
Submitted 6 October, 2018;
originally announced October 2018.
-
Fast Object Learning and Dual-arm Coordination for Cluttered Stowing, Picking, and Packing
Authors:
Max Schwarz,
Christian Lenz,
Germán Martín García,
Seongyong Koo,
Arul Selvam Periyasamy,
Michael Schreiber,
Sven Behnke
Abstract:
Robotic picking from cluttered bins is a demanding task, for which Amazon Robotics holds challenges. The 2017 Amazon Robotics Challenge (ARC) required stowing items into a storage system, picking specific items, and packing them into boxes. In this paper, we describe the entry of team NimbRo Picking. Our deep object perception pipeline can be quickly and efficiently adapted to new items using a cu…
▽ More
Robotic picking from cluttered bins is a demanding task, for which Amazon Robotics holds challenges. The 2017 Amazon Robotics Challenge (ARC) required stowing items into a storage system, picking specific items, and packing them into boxes. In this paper, we describe the entry of team NimbRo Picking. Our deep object perception pipeline can be quickly and efficiently adapted to new items using a custom turntable capture system and transfer learning. It produces high-quality item segments, on which grasp poses are found. A planning component coordinates manipulation actions between two robot arms, minimizing execution time. The system has been demonstrated successfully at ARC, where our team reached second places in both the picking task and the final stow-and-pick task. We also evaluate individual components.
△ Less
Submitted 6 October, 2018;
originally announced October 2018.
-
NimbRo Rescue: Solving Disaster-Response Tasks through Mobile Manipulation Robot Momaro
Authors:
Max Schwarz,
Tobias Rodehutskors,
David Droeschel,
Marius Beul,
Michael Schreiber,
Nikita Araslanov,
Ivan Ivanov,
Christian Lenz,
Jan Razlaw,
Sebastian Schüller,
David Schwarz,
Angeliki Topalidou-Kyniazopoulou,
Sven Behnke
Abstract:
Robots that solve complex tasks in environments too dangerous for humans to enter are desperately needed, e.g. for search and rescue applications. We describe our mobile manipulation robot Momaro, with which we participated successfully in the DARPA Robotics Challenge. It features a unique locomotion design with four legs ending in steerable wheels, which allows it both to drive omnidirectionally…
▽ More
Robots that solve complex tasks in environments too dangerous for humans to enter are desperately needed, e.g. for search and rescue applications. We describe our mobile manipulation robot Momaro, with which we participated successfully in the DARPA Robotics Challenge. It features a unique locomotion design with four legs ending in steerable wheels, which allows it both to drive omnidirectionally and to step over obstacles or climb. Furthermore, we present advanced communication and teleoperation approaches, which include immersive 3D visualization, and 6D tracking of operator head and arm motions. The proposed system is evaluated in the DARPA Robotics Challenge, the DLR SpaceBot Cup Qualification and lab experiments. We also discuss the lessons learned from the competitions.
△ Less
Submitted 15 October, 2018; v1 submitted 2 October, 2018;
originally announced October 2018.
-
Supervised Autonomous Locomotion and Manipulation for Disaster Response with a Centaur-like Robot
Authors:
Tobias Klamt,
Diego Rodriguez,
Max Schwarz,
Christian Lenz,
Dmytro Pavlichenko,
David Droeschel,
Sven Behnke
Abstract:
Mobile manipulation tasks are one of the key challenges in the field of search and rescue (SAR) robotics requiring robots with flexible locomotion and manipulation abilities. Since the tasks are mostly unknown in advance, the robot has to adapt to a wide variety of terrains and workspaces during a mission. The centaur-like robot Centauro has a hybrid legged-wheeled base and an anthropomorphic uppe…
▽ More
Mobile manipulation tasks are one of the key challenges in the field of search and rescue (SAR) robotics requiring robots with flexible locomotion and manipulation abilities. Since the tasks are mostly unknown in advance, the robot has to adapt to a wide variety of terrains and workspaces during a mission. The centaur-like robot Centauro has a hybrid legged-wheeled base and an anthropomorphic upper body to carry out complex tasks in environments too dangerous for humans. Due to its high number of degrees of freedom, controlling the robot with direct teleoperation approaches is challenging and exhausting. Supervised autonomy approaches are promising to increase quality and speed of control while keeping the flexibility to solve unknown tasks. We developed a set of operator assistance functionalities with different levels of autonomy to control the robot for challenging locomotion and manipulation tasks. The integrated system was evaluated in disaster response scenarios and showed promising performance.
△ Less
Submitted 6 March, 2019; v1 submitted 18 September, 2018;
originally announced September 2018.
-
A Non-Technical Survey on Deep Convolutional Neural Network Architectures
Authors:
Felix Altenberger,
Claus Lenz
Abstract:
Artificial neural networks have recently shown great results in many disciplines and a variety of applications, including natural language understanding, speech processing, games and image data generation. One particular application in which the strong performance of artificial neural networks was demonstrated is the recognition of objects in images, where deep convolutional neural networks are co…
▽ More
Artificial neural networks have recently shown great results in many disciplines and a variety of applications, including natural language understanding, speech processing, games and image data generation. One particular application in which the strong performance of artificial neural networks was demonstrated is the recognition of objects in images, where deep convolutional neural networks are commonly applied. In this survey, we give a comprehensive introduction to this topic (object recognition with deep convolutional neural networks), with a strong focus on the evolution of network architectures. Therefore, we aim to compress the most important concepts in this field in a simple and non-technical manner to allow for future researchers to have a quick general understanding.
This work is structured as follows:
1. We will explain the basic ideas of (convolutional) neural networks and deep learning and examine their usage for three object recognition tasks: image classification, object localization and object detection.
2. We give a review on the evolution of deep convolutional neural networks by providing an extensive overview of the most important network architectures presented in chronological order of their appearances.
△ Less
Submitted 6 March, 2018;
originally announced March 2018.
