-
Arena 3.0: Advancing Social Navigation in Collaborative and Highly Dynamic Environments
Authors:
Linh Kästner,
Volodymyir Shcherbyna,
Huajian Zeng,
Tuan Anh Le,
Maximilian Ho-Kyoung Schreff,
Halid Osmaev,
Nam Truong Tran,
Diego Diaz,
Jan Golebiowski,
Harold Soh,
Jens Lambrecht
Abstract:
Building upon our previous contributions, this paper introduces Arena 3.0, an extension of Arena-Bench, Arena 1.0, and Arena 2.0. Arena 3.0 is a comprehensive software stack containing multiple modules and simulation environments focusing on the development, simulation, and benchmarking of social navigation approaches in collaborative environments. We significantly enhance the realism of human beh…
▽ More
Building upon our previous contributions, this paper introduces Arena 3.0, an extension of Arena-Bench, Arena 1.0, and Arena 2.0. Arena 3.0 is a comprehensive software stack containing multiple modules and simulation environments focusing on the development, simulation, and benchmarking of social navigation approaches in collaborative environments. We significantly enhance the realism of human behavior simulation by incorporating a diverse array of new social force models and interaction patterns, encompassing both human-human and human-robot dynamics. The platform provides a comprehensive set of new task modes, designed for extensive benchmarking and testing and is capable of generating realistic and human-centric environments dynamically, catering to a broad spectrum of social navigation scenarios. In addition, the platform's functionalities have been abstracted across three widely used simulators, each tailored for specific training and testing purposes. The platform's efficacy has been validated through an extensive benchmark and user evaluations of the platform by a global community of researchers and students, which noted the substantial improvement compared to previous versions and expressed interests to utilize the platform for future research and development. Arena 3.0 is openly available at https://github.com/Arena-Rosnav.
△ Less
Submitted 2 June, 2024;
originally announced June 2024.
-
Mono Video-Based AI Corridor for Model-Free Detection of Collision-Relevant Obstacles
Authors:
Thomas Michalke,
Yassin Kaddar,
Thomas Nürnberg,
Linh Kästner,
Jens Lambrecht
Abstract:
The detection of previously unseen, unexpected obstacles on the road is a major challenge for automated driving systems. Different from the detection of ordinary objects with pre-definable classes, detecting unexpected obstacles on the road cannot be resolved by upscaling the sensor technology alone (e.g., high resolution video imagers / radar antennas, denser LiDAR scan lines). This is due to the…
▽ More
The detection of previously unseen, unexpected obstacles on the road is a major challenge for automated driving systems. Different from the detection of ordinary objects with pre-definable classes, detecting unexpected obstacles on the road cannot be resolved by upscaling the sensor technology alone (e.g., high resolution video imagers / radar antennas, denser LiDAR scan lines). This is due to the fact, that there is a wide variety in the types of unexpected obstacles that also do not share a common appearance (e.g., lost cargo as a suitcase or bicycle, tire fragments, a tree stem). Also adding object classes or adding \enquote{all} of these objects to a common \enquote{unexpected obstacle} class does not scale. In this contribution, we study the feasibility of using a deep learning video-based lane corridor (called \enquote{AI ego-corridor}) to ease the challenge by inverting the problem: Instead of detecting a previously unseen object, the AI ego-corridor detects that the ego-lane ahead ends. A smart ground-truth definition enables an easy feature-based classification of an abrupt end of the ego-lane. We propose two neural network designs and research among other things the potential of training with synthetic data. We evaluate our approach on a test vehicle platform. It is shown that the approach is able to detect numerous previously unseen obstacles at a distance of up to 300 m with a detection rate of 95 %.
△ Less
Submitted 8 May, 2023; v1 submitted 24 April, 2023;
originally announced April 2023.
-
HabitatDyn Dataset: Dynamic Object Detection to Kinematics Estimation
Authors:
Zhengcheng Shen,
Yi Gao,
Linh Kästner,
Jens Lambrecht
Abstract:
The advancement of computer vision and machine learning has made datasets a crucial element for further research and applications. However, the creation and development of robots with advanced recognition capabilities are hindered by the lack of appropriate datasets. Existing image or video processing datasets are unable to accurately depict observations from a moving robot, and they do not contai…
▽ More
The advancement of computer vision and machine learning has made datasets a crucial element for further research and applications. However, the creation and development of robots with advanced recognition capabilities are hindered by the lack of appropriate datasets. Existing image or video processing datasets are unable to accurately depict observations from a moving robot, and they do not contain the kinematics information necessary for robotic tasks. Synthetic data, on the other hand, are cost-effective to create and offer greater flexibility for adapting to various applications. Hence, they are widely utilized in both research and industry. In this paper, we propose the dataset HabitatDyn, which contains both synthetic RGB videos, semantic labels, and depth information, as well as kinetics information. HabitatDyn was created from the perspective of a mobile robot with a moving camera, and contains 30 scenes featuring six different types of moving objects with varying velocities. To demonstrate the usability of our dataset, two existing algorithms are used for evaluation and an approach to estimate the distance between the object and camera is implemented based on these segmentation methods and evaluated through the dataset. With the availability of this dataset, we aspire to foster further advancements in the field of mobile robotics, leading to more capable and intelligent robots that can navigate and interact with their environments more effectively. The code is publicly available at https://github.com/ignc-research/HabitatDyn.
△ Less
Submitted 21 April, 2023;
originally announced April 2023.
-
Arena-Rosnav 2.0: A Development and Benchmarking Platform for Robot Navigation in Highly Dynamic Environments
Authors:
Linh Kästner,
Reyk Carstens,
Huajian Zeng,
Jacek Kmiecik,
Teham Bhuiyan,
Niloufar Khorsandi,
Volodymyr Shcherbyna,
Jens Lambrecht
Abstract:
Following up on our previous works, in this paper, we present Arena-Rosnav 2.0 an extension to our previous works Arena-Bench and Arena-Rosnav, which adds a variety of additional modules for developing and benchmarking robotic navigation approaches. The platform is fundamentally restructured and provides unified APIs to add additional functionalities such as planning algorithms, simulators, or eva…
▽ More
Following up on our previous works, in this paper, we present Arena-Rosnav 2.0 an extension to our previous works Arena-Bench and Arena-Rosnav, which adds a variety of additional modules for developing and benchmarking robotic navigation approaches. The platform is fundamentally restructured and provides unified APIs to add additional functionalities such as planning algorithms, simulators, or evaluation functionalities. We have included more realistic simulation and pedestrian behavior and provide a profound documentation to lower the entry barrier. We evaluated our system by first, conducting a user study in which we asked experienced researchers as well as new practitioners and students to test our system. The feedback was mostly positive and a high number of participants are utilizing our system for other research endeavors. Finally, we demonstrate the feasibility of our system by integrating two new simulators and a variety of state of the art navigation approaches and benchmark them against one another. The platform is openly available at https://github.com/Arena-Rosnav.
△ Less
Submitted 31 July, 2023; v1 submitted 20 February, 2023;
originally announced February 2023.
-
Holistic Deep-Reinforcement-Learning-based Training of Autonomous Navigation Systems
Authors:
Linh Kästner,
Marvin Meusel,
Teham Bhuiyan,
Jens Lambrecht
Abstract:
In recent years, Deep Reinforcement Learning emerged as a promising approach for autonomous navigation of ground vehicles and has been utilized in various areas of navigation such as cruise control, lane changing, or obstacle avoidance. However, most research works either focus on providing an end-to-end solution training the whole system using Deep Reinforcement Learning or focus on one specific…
▽ More
In recent years, Deep Reinforcement Learning emerged as a promising approach for autonomous navigation of ground vehicles and has been utilized in various areas of navigation such as cruise control, lane changing, or obstacle avoidance. However, most research works either focus on providing an end-to-end solution training the whole system using Deep Reinforcement Learning or focus on one specific aspect such as local motion planning. This however, comes along with a number of problems such as catastrophic forgetfulness, inefficient navigation behavior, and non-optimal synchronization between different entities of the navigation stack. In this paper, we propose a holistic Deep Reinforcement Learning training approach in which the training procedure is involving all entities of the navigation stack. This should enhance the synchronization between- and understanding of all entities of the navigation stack and as a result, improve navigational performance. We trained several agents with a number of different observation spaces to study the impact of different input on the navigation behavior of the agent. In profound evaluations against multiple learning-based and classic model-based navigation approaches, our proposed agent could outperform the baselines in terms of efficiency and safety attaining shorter path lengths, less roundabout paths, and less collisions.
△ Less
Submitted 6 February, 2023;
originally announced February 2023.
-
Arena-Web -- A Web-based Development and Benchmarking Platform for Autonomous Navigation Approaches
Authors:
Linh Kästner,
Reyk Carstens,
Christopher Liebig,
Volodymyr Shcherbyna,
Lena Nahrworld,
Subhin Lee,
Jens Lambrecht
Abstract:
In recent years, mobile robot navigation approaches have become increasingly important due to various application areas ranging from healthcare to warehouse logistics. In particular, Deep Reinforcement Learning approaches have gained popularity for robot navigation but are not easily accessible to non-experts and complex to develop. In recent years, efforts have been made to make these sophisticat…
▽ More
In recent years, mobile robot navigation approaches have become increasingly important due to various application areas ranging from healthcare to warehouse logistics. In particular, Deep Reinforcement Learning approaches have gained popularity for robot navigation but are not easily accessible to non-experts and complex to develop. In recent years, efforts have been made to make these sophisticated approaches accessible to a wider audience. In this paper, we present Arena-Web, a web-based development and evaluation suite for developing, training, and testing DRL-based navigation planners for various robotic platforms and scenarios. The interface is designed to be intuitive and engaging to appeal to non-experts and make the technology accessible to a wider audience. With Arena-Web and its interface, training and developing Deep Reinforcement Learning agents is simplified and made easy without a single line of code. The web-app is free to use and openly available under the link stated in the supplementary materials.
△ Less
Submitted 6 February, 2023;
originally announced February 2023.
-
Evaluating Digital Work Instructions with Augmented Reality versus Paper-based Documents for Manual, Object-Specific Repair Tasks in a Case Study with Experienced Workers
Authors:
Leon Eversberg,
Jens Lambrecht
Abstract:
Manual repair tasks in the industry of maintenance, repair, and overhaul require experience and object-specific information. Today, many of these repair tasks are still performed and documented with inefficient paper documents. Cognitive assistance systems have the potential to reduce costs, errors, and mental workload by providing all required information digitally. In this case study, we present…
▽ More
Manual repair tasks in the industry of maintenance, repair, and overhaul require experience and object-specific information. Today, many of these repair tasks are still performed and documented with inefficient paper documents. Cognitive assistance systems have the potential to reduce costs, errors, and mental workload by providing all required information digitally. In this case study, we present an assistance system for object-specific repair tasks for turbine blades. The assistance system provides digital work instructions and uses augmented reality to display spatial information. In a user study with ten experienced metalworkers performing a familiar repair task, we compare time to task completion, subjective workload, and system usability of the new assistance system to their established paper-based workflow. All participants stated that they preferred the assistance system over the paper documents. The results of the study show that the manual repair task can be completed 21 % faster and with a 26 % lower perceived workload using the assistance system.
△ Less
Submitted 6 June, 2023; v1 submitted 18 January, 2023;
originally announced January 2023.
-
Deep-Reinforcement-Learning-based Path Planning for Industrial Robots using Distance Sensors as Observation
Authors:
Teham Bhuiyan,
Linh Kästner,
Yifan Hu,
Benno Kutschank,
Jens Lambrecht
Abstract:
Industrial robots are widely used in various manufacturing environments due to their efficiency in doing repetitive tasks such as assembly or welding. A common problem for these applications is to reach a destination without colliding with obstacles or other robot arms. Commonly used sampling-based path planning approaches such as RRT require long computation times, especially in complex environme…
▽ More
Industrial robots are widely used in various manufacturing environments due to their efficiency in doing repetitive tasks such as assembly or welding. A common problem for these applications is to reach a destination without colliding with obstacles or other robot arms. Commonly used sampling-based path planning approaches such as RRT require long computation times, especially in complex environments. Furthermore, the environment in which they are employed needs to be known beforehand. When utilizing the approaches in new environments, a tedious engineering effort in setting hyperparameters needs to be conducted, which is time- and cost-intensive. On the other hand, Deep Reinforcement Learning has shown remarkable results in dealing with unknown environments, generalizing new problem instances, and solving motion planning problems efficiently. On that account, this paper proposes a Deep-Reinforcement-Learning-based motion planner for robotic manipulators. We evaluated our model against state-of-the-art sampling-based planners in several experiments. The results show the superiority of our planner in terms of path length and execution time.
△ Less
Submitted 14 January, 2023;
originally announced January 2023.
-
Imagination-augmented Navigation Based on 2D Laser Sensor Observations
Authors:
Zhengcheng Shen,
Linh Kästner,
Magdalena Yordanova,
Jens Lambrecht
Abstract:
Autonomous navigation of mobile robots is an essential task for various industries. Sensor data is crucial to ensure safe and reliable navigation. However, sensor observations are often limited by different factors. Imagination can assist to enhance the view and aid navigation in dangerous or unknown situations where only limited sensor observation is available. In this paper, we propose an imagin…
▽ More
Autonomous navigation of mobile robots is an essential task for various industries. Sensor data is crucial to ensure safe and reliable navigation. However, sensor observations are often limited by different factors. Imagination can assist to enhance the view and aid navigation in dangerous or unknown situations where only limited sensor observation is available. In this paper, we propose an imagination-enhanced navigation based on 2D semantic laser scan data. The system contains an imagination module, which can predict the entire occupied area of the object. The imagination module is trained in a supervised manner using a collected training dataset from a 2D simulator. Four different imagination models are trained, and the imagination results are evaluated. Subsequently, the imagination results are integrated into the local and global cost map to benefit the navigation procedure. The approach is validated on three different test maps, with seven different paths for each map. The quality and numeric results showed that the agent with the imagination module could generate more reliable paths without passing beneath the object, with the cost of a longer path and slower velocity.
△ Less
Submitted 12 June, 2022;
originally announced June 2022.
-
Human-Following and -guiding in Crowded Environments using Semantic Deep-Reinforcement-Learning for Mobile Service Robots
Authors:
Linh Kästner,
Bassel Fatloun,
Zhengcheng Shen,
Daniel Gawrisch,
Jens Lambrecht
Abstract:
Assistance robots have gained widespread attention in various industries such as logistics and human assistance. The tasks of guiding or following a human in a crowded environment such as airports or train stations to carry weight or goods is still an open problem. In these use cases, the robot is not only required to intelligently interact with humans, but also to navigate safely among crowds. Th…
▽ More
Assistance robots have gained widespread attention in various industries such as logistics and human assistance. The tasks of guiding or following a human in a crowded environment such as airports or train stations to carry weight or goods is still an open problem. In these use cases, the robot is not only required to intelligently interact with humans, but also to navigate safely among crowds. Thus, especially highly dynamic environments pose a grand challenge due to the volatile behavior patterns and unpredictable movements of humans. In this paper, we propose a Deep-Reinforcement-Learning-based agent for human-guiding and -following tasks in crowded environments. Therefore, we incorporate semantic information to provide the agent with high-level information like the social states of humans, safety models, and class types. We evaluate our proposed approach against a benchmark approach without semantic information and demonstrated enhanced navigational safety and robustness. Moreover, we demonstrate that the agent could learn to adapt its behavior to humans, which improves the human-robot interaction significantly.
△ Less
Submitted 12 June, 2022;
originally announced June 2022.
-
Arena-Bench: A Benchmarking Suite for Obstacle Avoidance Approaches in Highly Dynamic Environments
Authors:
Linh Kästner,
Teham Bhuiyan,
Tuan Anh Le,
Elias Treis,
Johannes Cox,
Boris Meinardus,
Jacek Kmiecik,
Reyk Carstens,
Duc Pichel,
Bassel Fatloun,
Niloufar Khorsandi,
Jens Lambrecht
Abstract:
The ability to autonomously navigate safely, especially within dynamic environments, is paramount for mobile robotics. In recent years, DRL approaches have shown superior performance in dynamic obstacle avoidance. However, these learning-based approaches are often developed in specially designed simulation environments and are hard to test against conventional planning approaches. Furthermore, the…
▽ More
The ability to autonomously navigate safely, especially within dynamic environments, is paramount for mobile robotics. In recent years, DRL approaches have shown superior performance in dynamic obstacle avoidance. However, these learning-based approaches are often developed in specially designed simulation environments and are hard to test against conventional planning approaches. Furthermore, the integration and deployment of these approaches into real robotic platforms are not yet completely solved. In this paper, we present Arena-bench, a benchmark suite to train, test, and evaluate navigation planners on different robotic platforms within 3D environments. It provides tools to design and generate highly dynamic evaluation worlds, scenarios, and tasks for autonomous navigation and is fully integrated into the robot operating system. To demonstrate the functionalities of our suite, we trained a DRL agent on our platform and compared it against a variety of existing different model-based and learning-based navigation approaches on a variety of relevant metrics. Finally, we deployed the approaches towards real robots and demonstrated the reproducibility of the results. The code is publicly available at github.com/ignc-research/arena-bench.
△ Less
Submitted 11 July, 2022; v1 submitted 12 June, 2022;
originally announced June 2022.
-
Obstacle-aware Waypoint Generation for Long-range Guidance of Deep-Reinforcement-Learning-based Navigation Approaches
Authors:
Linh Kästner,
Xinlin Zhao,
Zhengcheng Shen,
Jens Lambrecht
Abstract:
Navigation of mobile robots within crowded environments is an essential task in various use cases, such as delivery, health care, or logistics. Deep Reinforcement Learning (DRL) emerged as an alternative method to replace overly conservative approaches and promises more efficient and flexible navigation. However, Deep Reinforcement Learning is limited to local navigation due to its myopic nature.…
▽ More
Navigation of mobile robots within crowded environments is an essential task in various use cases, such as delivery, health care, or logistics. Deep Reinforcement Learning (DRL) emerged as an alternative method to replace overly conservative approaches and promises more efficient and flexible navigation. However, Deep Reinforcement Learning is limited to local navigation due to its myopic nature. Previous research works proposed various ways to combine Deep Reinforcement Learning with conventional methods but a common problem is the complexity of highly dynamic environments due to the unpredictability of humans and other objects within the environment. In this paper, we propose a hierarchical waypoint generator, which considers moving obstacles and thus generates safer and more robust waypoints for Deep-Reinforcement-Learning-based local planners. Therefore, we utilize Delaunay Triangulation to encode obstacles and incorporate an extended hybrid A-Star approach to efficiently search for an optimal solution in the time-state space. We compared our waypoint generator against two baseline approaches and outperform them in terms of safety, efficiency, and robustness.
△ Less
Submitted 23 September, 2021;
originally announced September 2021.
-
All-in-One: A DRL-based Control Switch Combining State-of-the-art Navigation Planners
Authors:
Linh Kästner,
Johannes Cox,
Teham Buiyan,
Jens Lambrecht
Abstract:
Autonomous navigation of mobile robots is an essential aspect in use cases such as delivery, assistance or logistics. Although traditional planning methods are well integrated into existing navigation systems, they struggle in highly dynamic environments. On the other hand, Deep-Reinforcement-Learning-based methods show superior performance in dynamic obstacle avoidance but are not suitable for lo…
▽ More
Autonomous navigation of mobile robots is an essential aspect in use cases such as delivery, assistance or logistics. Although traditional planning methods are well integrated into existing navigation systems, they struggle in highly dynamic environments. On the other hand, Deep-Reinforcement-Learning-based methods show superior performance in dynamic obstacle avoidance but are not suitable for long-range navigation and struggle with local minima. In this paper, we propose a Deep-Reinforcement-Learning-based control switch, which has the ability to select between different planning paradigms based solely on sensor data observations. Therefore, we develop an interface to efficiently operate multiple model-based, as well as learning-based local planners and integrate a variety of state-of-the-art planners to be selected by the control switch. Subsequently, we evaluate our approach against each planner individually and found improvements in navigation performance especially for highly dynamic scenarios. Our planner was able to prefer learning-based approaches in situations with a high number of obstacles while relying on the traditional model-based planners in long corridors or empty spaces.
△ Less
Submitted 23 September, 2021;
originally announced September 2021.
-
Enhancing Navigational Safety in Crowded Environments using Semantic-Deep-Reinforcement-Learning-based Navigation
Authors:
Linh Kästner,
Junhui Li,
Zhengcheng Shen,
Jens Lambrecht
Abstract:
Intelligent navigation among social crowds is an essential aspect of mobile robotics for applications such as delivery, health care, or assistance. Deep Reinforcement Learning emerged as an alternative planning method to conservative approaches and promises more efficient and flexible navigation. However, in highly dynamic environments employing different kinds of obstacle classes, safe navigation…
▽ More
Intelligent navigation among social crowds is an essential aspect of mobile robotics for applications such as delivery, health care, or assistance. Deep Reinforcement Learning emerged as an alternative planning method to conservative approaches and promises more efficient and flexible navigation. However, in highly dynamic environments employing different kinds of obstacle classes, safe navigation still presents a grand challenge. In this paper, we propose a semantic Deep-reinforcement-learning-based navigation approach that teaches object-specific safety rules by considering high-level obstacle information. In particular, the agent learns object-specific behavior by contemplating the specific danger zones to enhance safety around vulnerable object classes. We tested the approach against a benchmark obstacle avoidance approach and found an increase in safety. Furthermore, we demonstrate that the agent could learn to navigate more safely by keeping an individual safety distance dependent on the semantic information.
△ Less
Submitted 23 September, 2021;
originally announced September 2021.
-
Connecting Deep-Reinforcement-Learning-based Obstacle Avoidance with Conventional Global Planners using Waypoint Generators
Authors:
Linh Kästner,
Teham Buiyan,
Xinlin Zhao,
Zhengcheng Shen,
Cornelius Marx,
Jens Lambrecht
Abstract:
Deep Reinforcement Learning has emerged as an efficient dynamic obstacle avoidance method in highly dynamic environments. It has the potential to replace overly conservative or inefficient navigation approaches. However, the integration of Deep Reinforcement Learning into existing navigation systems is still an open frontier due to the myopic nature of Deep-Reinforcement-Learning-based navigation,…
▽ More
Deep Reinforcement Learning has emerged as an efficient dynamic obstacle avoidance method in highly dynamic environments. It has the potential to replace overly conservative or inefficient navigation approaches. However, the integration of Deep Reinforcement Learning into existing navigation systems is still an open frontier due to the myopic nature of Deep-Reinforcement-Learning-based navigation, which hinders its widespread integration into current navigation systems. In this paper, we propose the concept of an intermediate planner to interconnect novel Deep-Reinforcement-Learning-based obstacle avoidance with conventional global planning methods using waypoint generation. Therefore, we integrate different waypoint generators into existing navigation systems and compare the joint system against traditional ones. We found an increased performance in terms of safety, efficiency and path smoothness especially in highly dynamic environments.
△ Less
Submitted 8 April, 2021;
originally announced April 2021.
-
Spatial Imagination With Semantic Cognition for Mobile Robots
Authors:
Zhengcheng Shen,
Linh Kästner,
Jens Lambrecht
Abstract:
The imagination of the surrounding environment based on experience and semantic cognition has great potential to extend the limited observations and provide more information for mapping, collision avoidance, and path planning. This paper provides a training-based algorithm for mobile robots to perform spatial imagination based on semantic cognition and evaluates the proposed method for the mapping…
▽ More
The imagination of the surrounding environment based on experience and semantic cognition has great potential to extend the limited observations and provide more information for mapping, collision avoidance, and path planning. This paper provides a training-based algorithm for mobile robots to perform spatial imagination based on semantic cognition and evaluates the proposed method for the mapping task. We utilize a photo-realistic simulation environment, Habitat, for training and evaluation. The trained model is composed of Resent-18 as encoder and Unet as the backbone. We demonstrate that the algorithm can perform imagination for unseen parts of the object universally, by recalling the images and experience and compare our approach with traditional semantic mapping methods. It is found that our approach will improve the efficiency and accuracy of semantic mapping.
△ Less
Submitted 8 April, 2021;
originally announced April 2021.
-
Arena-Rosnav: Towards Deployment of Deep-Reinforcement-Learning-Based Obstacle Avoidance into Conventional Autonomous Navigation Systems
Authors:
Linh Kästner,
Teham Buiyan,
Xinlin Zhao,
Lei Jiao,
Zhengcheng Shen,
Jens Lambrecht
Abstract:
Recently, mobile robots have become important tools in various industries, especially in logistics. Deep reinforcement learning emerged as an alternative planning method to replace overly conservative approaches and promises more efficient and flexible navigation. However, deep reinforcement learning approaches are not suitable for long-range navigation due to their proneness to local minima and l…
▽ More
Recently, mobile robots have become important tools in various industries, especially in logistics. Deep reinforcement learning emerged as an alternative planning method to replace overly conservative approaches and promises more efficient and flexible navigation. However, deep reinforcement learning approaches are not suitable for long-range navigation due to their proneness to local minima and lack of long term memory, which hinders its widespread integration into industrial applications of mobile robotics. In this paper, we propose a navigation system incorporating deep-reinforcement-learning-based local planners into conventional navigation stacks for long-range navigation. Therefore, a framework for training and testing the deep reinforcement learning algorithms along with classic approaches is presented. We evaluated our deep-reinforcement-learning-enhanced navigation system against various conventional planners and found that our system outperforms them in terms of safety, efficiency and robustness.
△ Less
Submitted 23 September, 2021; v1 submitted 8 April, 2021;
originally announced April 2021.
-
Classification of Spot-welded Joints in Laser Thermography Data using Convolutional Neural Networks
Authors:
Linh Kästner,
Samim Ahmadi,
Florian Jonietz,
Mathias Ziegler,
Peter Jung,
Giuseppe Caire,
Jens Lambrecht
Abstract:
Spot welding is a crucial process step in various industries. However, classification of spot welding quality is still a tedious process due to the complexity and sensitivity of the test material, which drain conventional approaches to its limits. In this paper, we propose an approach for quality inspection of spot weldings using images from laser thermography data.We propose data preparation appr…
▽ More
Spot welding is a crucial process step in various industries. However, classification of spot welding quality is still a tedious process due to the complexity and sensitivity of the test material, which drain conventional approaches to its limits. In this paper, we propose an approach for quality inspection of spot weldings using images from laser thermography data.We propose data preparation approaches based on the underlying physics of spot welded joints, heated with pulsed laser thermography by analyzing the intensity over time and derive dedicated data filters to generate training datasets. Subsequently, we utilize convolutional neural networks to classify weld quality and compare the performance of different models against each other. We achieve competitive results in terms of classifying the different welding quality classes compared to traditional approaches, reaching an accuracy of more than 95 percent. Finally, we explore the effect of different augmentation methods.
△ Less
Submitted 24 October, 2020;
originally announced October 2020.
-
Integrative Object and Pose to Task Detection for an Augmented-Reality-based Human Assistance System using Neural Networks
Authors:
Linh Kästner,
Leon Eversberg,
Marina Mursa,
Jens Lambrecht
Abstract:
As a result of an increasingly automatized and digitized industry, processes are becoming more complex. Augmented Reality has shown considerable potential in assisting workers with complex tasks by enhancing user understanding and experience with spatial information. However, the acceptance and integration of AR into industrial processes is still limited due to the lack of established methods and…
▽ More
As a result of an increasingly automatized and digitized industry, processes are becoming more complex. Augmented Reality has shown considerable potential in assisting workers with complex tasks by enhancing user understanding and experience with spatial information. However, the acceptance and integration of AR into industrial processes is still limited due to the lack of established methods and tedious integration efforts. Meanwhile, deep neural networks have achieved remarkable results in computer vision tasks and bear great prospects to enrich Augmented Reality applications . In this paper, we propose an Augmented-Reality-based human assistance system to assist workers in complex manual tasks where we incorporate deep neural networks for computer vision tasks. More specifically, we combine Augmented Reality with object and action detectors to make workflows more intuitive and flexible. To evaluate our system in terms of user acceptance and efficiency, we conducted several user studies. We found a significant reduction in time to task completion in untrained workers and a decrease in error rate. Furthermore, we investigated the users learning curve with our assistance system.
△ Less
Submitted 31 August, 2020;
originally announced August 2020.
-
Deep-Reinforcement-Learning-Based Semantic Navigation of Mobile Robots in Dynamic Environments
Authors:
Linh Kästner,
Cornelius Marx,
Jens Lambrecht
Abstract:
Mobile robots have gained increased importance within industrial tasks such as commissioning, delivery or operation in hazardous environments. The ability to autonomously navigate safely especially within dynamic environments, is paramount in industrial mobile robotics. Current navigation methods depend on preexisting static maps and are error-prone in dynamic environments. Furthermore, for safety…
▽ More
Mobile robots have gained increased importance within industrial tasks such as commissioning, delivery or operation in hazardous environments. The ability to autonomously navigate safely especially within dynamic environments, is paramount in industrial mobile robotics. Current navigation methods depend on preexisting static maps and are error-prone in dynamic environments. Furthermore, for safety reasons, they often rely on hand-crafted safety guidelines, which makes the system less flexible and slow. Visual based navigation and high level semantics bear the potential to enhance the safety of path planing by creating links the agent can reason about for a more flexible navigation. On this account, we propose a reinforcement learning based local navigation system which learns navigation behavior based solely on visual observations to cope with highly dynamic environments. Therefore, we develop a simple yet efficient simulator - ARENA2D - which is able to generate highly randomized training environments and provide semantic information to train our agent. We demonstrate enhanced results in terms of safety and robustness over a traditional baseline approach based on the dynamic window approach.
△ Less
Submitted 2 August, 2020;
originally announced August 2020.
-
A Markerless Deep Learning-based 6 Degrees of Freedom PoseEstimation for with Mobile Robots using RGB Data
Authors:
Linh Kästner,
Daniel Dimitrov,
Jens Lambrecht
Abstract:
Augmented Reality has been subject to various integration efforts within industries due to its ability to enhance human machine interaction and understanding. Neural networks have achieved remarkable results in areas of computer vision, which bear great potential to assist and facilitate an enhanced Augmented Reality experience. However, most neural networks are computationally intensive and deman…
▽ More
Augmented Reality has been subject to various integration efforts within industries due to its ability to enhance human machine interaction and understanding. Neural networks have achieved remarkable results in areas of computer vision, which bear great potential to assist and facilitate an enhanced Augmented Reality experience. However, most neural networks are computationally intensive and demand huge processing power thus, are not suitable for deployment on Augmented Reality devices. In this work we propose a method to deploy state of the art neural networks for real time 3D object localization on augmented reality devices. As a result, we provide a more automated method of calibrating the AR devices with mobile robotic systems. To accelerate the calibration process and enhance user experience, we focus on fast 2D detection approaches which are extracting the 3D pose of the object fast and accurately by using only 2D input. The results are implemented into an Augmented Reality application for intuitive robot control and sensor data visualization. For the 6D annotation of 2D images, we developed an annotation tool, which is, to our knowledge, the first open source tool to be available. We achieve feasible results which are generally applicable to any AR device thus making this work promising for further research in combining high demanding neural networks with Internet of Things devices.
△ Less
Submitted 16 January, 2020;
originally announced January 2020.
-
Augmented-Reality-Based Visualization of Navigation Data of Mobile Robots on the Microsoft Hololens -- Possibilities and Limitations
Authors:
Linh Kästner,
Jens Lambrecht
Abstract:
The demand for mobile robots has rapidly increased in recent years due to the flexibility and high variety of application fields comparing to static robots. To deal with complex tasks such as navigation, they work with high amounts of different sensor data making it difficult to operate with for non-experts. To enhance user understanding and human robot interaction, we propose an approach to visua…
▽ More
The demand for mobile robots has rapidly increased in recent years due to the flexibility and high variety of application fields comparing to static robots. To deal with complex tasks such as navigation, they work with high amounts of different sensor data making it difficult to operate with for non-experts. To enhance user understanding and human robot interaction, we propose an approach to visualize the navigation stack within a cutting edge 3D Augmented Reality device -- the Microsoft Hololens. Therefore, relevant navigation stack data including laser scan, environment map and path planing data are visualized in 3D within the head mounted device. Based on that prototype, we evaluate the Hololens in terms of computational capabilities and limitations for dealing with huge amount of real-time data. Results show that the Hololens is capable of a proper visualization of huge amounts of sensor data. We demonstrate a proper visualization of navigation stack data in 3D within the Hololens. However, there are limitations when transferring and displaying different kinds of data simultaneously.
△ Less
Submitted 27 December, 2019;
originally announced December 2019.
-
A 3D-Deep-Learning-based Augmented Reality Calibration Method for Robotic Environments using Depth Sensor Data
Authors:
Linh Kästner,
Vlad Catalin Frasineanu,
Jens Lambrecht
Abstract:
Augmented Reality and mobile robots are gaining much attention within industries due to the high potential to make processes cost and time efficient. To facilitate augmented reality, a calibration between the Augmented Reality device and the environment is necessary. This is a challenge when dealing with mobile robots due to the mobility of all entities making the environment dynamic. On this acco…
▽ More
Augmented Reality and mobile robots are gaining much attention within industries due to the high potential to make processes cost and time efficient. To facilitate augmented reality, a calibration between the Augmented Reality device and the environment is necessary. This is a challenge when dealing with mobile robots due to the mobility of all entities making the environment dynamic. On this account, we propose a novel approach to calibrate the Augmented Reality device using 3D depth sensor data. We use the depth camera of a cutting edge Augmented Reality Device - the Microsoft Hololens for deep learning based calibration. Therefore, we modified a neural network based on the recently published VoteNet architecture which works directly on the point cloud input observed by the Hololens. We achieve satisfying results and eliminate external tools like markers, thus enabling a more intuitive and flexible work flow for Augmented Reality integration. The results are adaptable to work with all depth cameras and are promising for further research. Furthermore, we introduce an open source 3D point cloud labeling tool, which is to our knowledge the first open source tool for labeling raw point cloud data.
△ Less
Submitted 27 December, 2019;
originally announced December 2019.
