-
Differentiable Collision-Free Parametric Corridors
Authors:
Jon Arrizabalaga,
Zachary Manchester,
Markus Ryll
Abstract:
This paper presents a method to compute differentiable collision-free parametric corridors. In contrast to existing solutions that decompose the obstacle-free space into multiple convex sets, the continuous corridors computed by our method are smooth and differentiable, making them compatible with existing numerical techniques for learning and optimization. To achieve this, we represent the collis…
▽ More
This paper presents a method to compute differentiable collision-free parametric corridors. In contrast to existing solutions that decompose the obstacle-free space into multiple convex sets, the continuous corridors computed by our method are smooth and differentiable, making them compatible with existing numerical techniques for learning and optimization. To achieve this, we represent the collision-free corridors as a path-parametric off-centered ellipse with a polynomial basis. We show that the problem of maximizing the volume of such corridors is convex, and can be efficiently solved. To assess the effectiveness of the proposed method, we examine its performance in a synthetic case study and subsequently evaluate its applicability in a real-world scenario from the KITTI dataset.
△ Less
Submitted 16 July, 2024;
originally announced July 2024.
-
Dual Quaternion Control of UAVs with Cable-suspended Load
Authors:
Yuxia Yuan,
Markus Ryll
Abstract:
Modeling the kinematics and dynamics of robotics systems with suspended loads using dual quaternions has not been explored so far. This paper introduces a new innovative control strategy using dual quaternions for UAVs with cable-suspended loads, focusing on the sling load lifting and tracking problems. By utilizing the mathematical efficiency and compactness of dual quaternions, a unified represe…
▽ More
Modeling the kinematics and dynamics of robotics systems with suspended loads using dual quaternions has not been explored so far. This paper introduces a new innovative control strategy using dual quaternions for UAVs with cable-suspended loads, focusing on the sling load lifting and tracking problems. By utilizing the mathematical efficiency and compactness of dual quaternions, a unified representation of the UAV and its suspended load's dynamics and kinematics is achieved, facilitating the realization of load lifting and trajectory tracking. The simulation results have tested the proposed strategy's accuracy, efficiency, and robustness. This study makes a substantial contribution to present this novel control strategy that harnesses the benefits of dual quaternions for cargo UAVs. Our work also holds promise for inspiring future innovations in under-actuated systems control using dual quaternions.
△ Less
Submitted 11 April, 2024;
originally announced April 2024.
-
Scene-Graph ViT: End-to-End Open-Vocabulary Visual Relationship Detection
Authors:
Tim Salzmann,
Markus Ryll,
Alex Bewley,
Matthias Minderer
Abstract:
Visual relationship detection aims to identify objects and their relationships in images. Prior methods approach this task by adding separate relationship modules or decoders to existing object detection architectures. This separation increases complexity and hinders end-to-end training, which limits performance. We propose a simple and highly efficient decoder-free architecture for open-vocabular…
▽ More
Visual relationship detection aims to identify objects and their relationships in images. Prior methods approach this task by adding separate relationship modules or decoders to existing object detection architectures. This separation increases complexity and hinders end-to-end training, which limits performance. We propose a simple and highly efficient decoder-free architecture for open-vocabulary visual relationship detection. Our model consists of a Transformer-based image encoder that represents objects as tokens and models their relationships implicitly. To extract relationship information, we introduce an attention mechanism that selects object pairs likely to form a relationship. We provide a single-stage recipe to train this model on a mixture of object and relationship detection data. Our approach achieves state-of-the-art relationship detection performance on Visual Genome and on the large-vocabulary GQA benchmark at real-time inference speeds. We provide ablations, real-world qualitative examples, and analyses of zero-shot performance.
△ Less
Submitted 19 July, 2024; v1 submitted 21 March, 2024;
originally announced March 2024.
-
Geometric Slosh-Free Tracking for Robotic Manipulators
Authors:
Jon Arrizabalaga,
Lukas Pries,
Riddhiman Laha,
Runkang Li,
Sami Haddadin,
Markus Ryll
Abstract:
This work focuses on the agile transportation of liquids with robotic manipulators. In contrast to existing methods that are either computationally heavy, system/container specific or dependant on a singularity-prone pendulum model, we present a real-time slosh-free tracking technique. This method solely requires the reference trajectory and the robot's kinematic constraints to output kinematicall…
▽ More
This work focuses on the agile transportation of liquids with robotic manipulators. In contrast to existing methods that are either computationally heavy, system/container specific or dependant on a singularity-prone pendulum model, we present a real-time slosh-free tracking technique. This method solely requires the reference trajectory and the robot's kinematic constraints to output kinematically feasible joint space commands. The crucial element underlying this approach consists on mimicking the end-effector's motion through a virtual quadrotor, which is inherently slosh-free and differentially flat, thereby allowing us to calculate a slosh-free reference orientation. Through the utilization of a cascaded proportional-derivative (PD) controller, this slosh-free reference is transformed into task space acceleration commands, which, following the resolution of a Quadratic Program (QP) based on Resolved Acceleration Control (RAC), are translated into a feasible joint configuration. The validity of the proposed approach is demonstrated by simulated and real-world experiments on a 7 DoF Franka Emika Panda robot.
Code: https://github.com/jonarriza96/gsft Video: https://youtu.be/4kitqYVS9n8
△ Less
Submitted 7 February, 2024;
originally announced February 2024.
-
Learning for CasADi: Data-driven Models in Numerical Optimization
Authors:
Tim Salzmann,
Jon Arrizabalaga,
Joel Andersson,
Marco Pavone,
Markus Ryll
Abstract:
While real-world problems are often challenging to analyze analytically, deep learning excels in modeling complex processes from data. Existing optimization frameworks like CasADi facilitate seamless usage of solvers but face challenges when integrating learned process models into numerical optimizations. To address this gap, we present the Learning for CasADi (L4CasADi) framework, enabling the se…
▽ More
While real-world problems are often challenging to analyze analytically, deep learning excels in modeling complex processes from data. Existing optimization frameworks like CasADi facilitate seamless usage of solvers but face challenges when integrating learned process models into numerical optimizations. To address this gap, we present the Learning for CasADi (L4CasADi) framework, enabling the seamless integration of PyTorch-learned models with CasADi for efficient and potentially hardware-accelerated numerical optimization. The applicability of L4CasADi is demonstrated with two tutorial examples: First, we optimize a fish's trajectory in a turbulent river for energy efficiency where the turbulent flow is represented by a PyTorch model. Second, we demonstrate how an implicit Neural Radiance Field environment representation can be easily leveraged for optimal control with L4CasADi. L4CasADi, along with examples and documentation, is available under MIT license at https://github.com/Tim-Salzmann/l4casadi
△ Less
Submitted 10 December, 2023;
originally announced December 2023.
-
Robots That Can See: Leveraging Human Pose for Trajectory Prediction
Authors:
Tim Salzmann,
Lewis Chiang,
Markus Ryll,
Dorsa Sadigh,
Carolina Parada,
Alex Bewley
Abstract:
Anticipating the motion of all humans in dynamic environments such as homes and offices is critical to enable safe and effective robot navigation. Such spaces remain challenging as humans do not follow strict rules of motion and there are often multiple occluded entry points such as corners and doors that create opportunities for sudden encounters. In this work, we present a Transformer based arch…
▽ More
Anticipating the motion of all humans in dynamic environments such as homes and offices is critical to enable safe and effective robot navigation. Such spaces remain challenging as humans do not follow strict rules of motion and there are often multiple occluded entry points such as corners and doors that create opportunities for sudden encounters. In this work, we present a Transformer based architecture to predict human future trajectories in human-centric environments from input features including human positions, head orientations, and 3D skeletal keypoints from onboard in-the-wild sensory information. The resulting model captures the inherent uncertainty for future human trajectory prediction and achieves state-of-the-art performance on common prediction benchmarks and a human tracking dataset captured from a mobile robot adapted for the prediction task. Furthermore, we identify new agents with limited historical data as a major contributor to error and demonstrate the complementary nature of 3D skeletal poses in reducing prediction error in such challenging scenarios.
△ Less
Submitted 29 September, 2023;
originally announced September 2023.
-
Pose-Following with Dual Quaternions
Authors:
Jon Arrizabalaga,
Markus Ryll
Abstract:
This work focuses on pose-following, a variant of path-following in which the goal is to steer the system's position and attitude along a path with a moving frame attached to it. Full body motion control, while accounting for the additional freedom to self-regulate the progress along the path, is an appealing trade-off. Towards this end, we extend the well-established dual quaternion-based pose-tr…
▽ More
This work focuses on pose-following, a variant of path-following in which the goal is to steer the system's position and attitude along a path with a moving frame attached to it. Full body motion control, while accounting for the additional freedom to self-regulate the progress along the path, is an appealing trade-off. Towards this end, we extend the well-established dual quaternion-based pose-tracking method into a pose-following control law. Specifically, we derive the equations of motion for the full pose error between the geometric reference and the rigid body in the form of a dual quaternion and dual twist. Subsequently, we formulate an almost globally asymptotically stable control law. The global attractivity of the presented approach is validated in a spatial example, while its benefits over pose-tracking are showcased through a planar case-study.
△ Less
Submitted 14 December, 2023; v1 submitted 18 August, 2023;
originally announced August 2023.
-
SCTOMP: Spatially Constrained Time-Optimal Motion Planning
Authors:
Jon Arrizabalaga,
Markus Ryll
Abstract:
This paper focuses on spatial time-optimal motion planning, a generalization of the exact time-optimal path following problem that allows the system to plan within a predefined space. In contrast to state-of-the-art methods, we drop the assumption that a collision-free geometric reference is given. Instead, we present a two-stage motion planning method that solely relies on a goal location and a g…
▽ More
This paper focuses on spatial time-optimal motion planning, a generalization of the exact time-optimal path following problem that allows the system to plan within a predefined space. In contrast to state-of-the-art methods, we drop the assumption that a collision-free geometric reference is given. Instead, we present a two-stage motion planning method that solely relies on a goal location and a geometric representation of the environment to compute a time-optimal trajectory that is compliant with system dynamics and constraints. To do so, the proposed scheme first computes an obstacle-free Pythagorean Hodograph parametric spline, and second solves a spatially reformulated minimum-time optimization problem. The spline obtained in the first stage is not a geometric reference, but an extension of the environment representation, and thus, time-optimality of the solution is guaranteed. The efficacy of the proposed approach is benchmarked by a known planar example and validated in a more complex spatial system, illustrating its versatility and applicability.
△ Less
Submitted 15 July, 2023; v1 submitted 5 October, 2022;
originally announced October 2022.
-
Spatial motion planning with Pythagorean Hodograph curves
Authors:
Jon Arrizabalaga,
Markus Ryll
Abstract:
This paper presents a two-stage prediction-based control scheme for embedding the environment's geometric properties into a collision-free Pythagorean Hodograph spline, and subsequently finding the optimal path within the parameterized free space. The ingredients of this approach are twofold: First, we present a novel spatial path parameterization applicable to any arbitrary curve without prior as…
▽ More
This paper presents a two-stage prediction-based control scheme for embedding the environment's geometric properties into a collision-free Pythagorean Hodograph spline, and subsequently finding the optimal path within the parameterized free space. The ingredients of this approach are twofold: First, we present a novel spatial path parameterization applicable to any arbitrary curve without prior assumptions in its adapted frame. Second, we identify the appropriateness of Pythagorean Hodograph curves for a compact and continuous definition of the path-parametric functions required by the presented spatial model. This dual-stage formulation results in a motion planning approach, where the geometric properties of the environment arise as states of the prediction model. Thus, the presented method is attractive for motion planning in dense environments. The efficacy of the approach is evaluated according to an illustrative example.
△ Less
Submitted 4 September, 2022;
originally announced September 2022.
-
Visual Navigation for Autonomous Vehicles: An Open-source Hands-on Robotics Course at MIT
Authors:
Luca Carlone,
Kasra Khosoussi,
Vasileios Tzoumas,
Golnaz Habibi,
Markus Ryll,
Rajat Talak,
Jingnan Shi,
Pasquale Antonante
Abstract:
This paper reports on the development, execution, and open-sourcing of a new robotics course at MIT. The course is a modern take on "Visual Navigation for Autonomous Vehicles" (VNAV) and targets first-year graduate students and senior undergraduates with prior exposure to robotics. VNAV has the goal of preparing the students to perform research in robotics and vision-based navigation, with emphasi…
▽ More
This paper reports on the development, execution, and open-sourcing of a new robotics course at MIT. The course is a modern take on "Visual Navigation for Autonomous Vehicles" (VNAV) and targets first-year graduate students and senior undergraduates with prior exposure to robotics. VNAV has the goal of preparing the students to perform research in robotics and vision-based navigation, with emphasis on drones and self-driving cars. The course spans the entire autonomous navigation pipeline; as such, it covers a broad set of topics, including geometric control and trajectory optimization, 2D and 3D computer vision, visual and visual-inertial odometry, place recognition, simultaneous localization and mapping, and geometric deep learning for perception. VNAV has three key features. First, it bridges traditional computer vision and robotics courses by exposing the challenges that are specific to embodied intelligence, e.g., limited computation and need for just-in-time and robust perception to close the loop over control and decision making. Second, it strikes a balance between depth and breadth by combining rigorous technical notes (including topics that are less explored in typical robotics courses, e.g., on-manifold optimization) with slides and videos showcasing the latest research results. Third, it provides a compelling approach to hands-on robotics education by leveraging a physical drone platform (mostly suitable for small residential courses) and a photo-realistic Unity-based simulator (open-source and scalable to large online courses). VNAV has been offered at MIT in the Falls of 2018-2021 and is now publicly available on MIT OpenCourseWare (OCW).
△ Less
Submitted 1 June, 2022;
originally announced June 2022.
-
SMORS: A soft multirotor UAV for multimodal locomotion and robust interaction
Authors:
Markus Ryll,
Robert K. Katzschmann
Abstract:
We present SMORS, the first Soft fully actuated MultirOtoR System for multimodal locomotion. Unlike conventional hexarotors, SMORS is equipped with three rigid and three continuously soft arms, with each arm hosting a propeller. We create a bridge between the fields of soft and aerial robotics by mechanically coupling the actuation of a fully actuated flying platform with the actuation of a soft r…
▽ More
We present SMORS, the first Soft fully actuated MultirOtoR System for multimodal locomotion. Unlike conventional hexarotors, SMORS is equipped with three rigid and three continuously soft arms, with each arm hosting a propeller. We create a bridge between the fields of soft and aerial robotics by mechanically coupling the actuation of a fully actuated flying platform with the actuation of a soft robotic manipulator. Each rotor is slightly tilted, allowing for full actuation of the platform. The soft components combined with the platform's full actuation allow for a robust interaction, in the form of efficient multimodal locomotion. In this work, we present the dynamical model of the platform, derive a closed-loop control, and present simulation results fortifying the robustness of the platform under a jumping-flying maneuver. We demonstrate in simulations that our multimodal locomotion approach can be more energy-efficient than the flight with a hexarotor.
△ Less
Submitted 5 April, 2022;
originally announced April 2022.
-
Real-time Neural-MPC: Deep Learning Model Predictive Control for Quadrotors and Agile Robotic Platforms
Authors:
Tim Salzmann,
Elia Kaufmann,
Jon Arrizabalaga,
Marco Pavone,
Davide Scaramuzza,
Markus Ryll
Abstract:
Model Predictive Control (MPC) has become a popular framework in embedded control for high-performance autonomous systems. However, to achieve good control performance using MPC, an accurate dynamics model is key. To maintain real-time operation, the dynamics models used on embedded systems have been limited to simple first-principle models, which substantially limits their representative power. I…
▽ More
Model Predictive Control (MPC) has become a popular framework in embedded control for high-performance autonomous systems. However, to achieve good control performance using MPC, an accurate dynamics model is key. To maintain real-time operation, the dynamics models used on embedded systems have been limited to simple first-principle models, which substantially limits their representative power. In contrast to such simple models, machine learning approaches, specifically neural networks, have been shown to accurately model even complex dynamic effects, but their large computational complexity hindered combination with fast real-time iteration loops. With this work, we present Real-time Neural MPC, a framework to efficiently integrate large, complex neural network architectures as dynamics models within a model-predictive control pipeline. Our experiments, performed in simulation and the real world onboard a highly agile quadrotor platform, demonstrate the capabilities of the described system to run learned models with, previously infeasible, large modeling capacity using gradient-based online optimization MPC. Compared to prior implementations of neural networks in online optimization MPC we can leverage models of over 4000 times larger parametric capacity in a 50Hz real-time window on an embedded platform. Further, we show the feasibility of our framework on real-world problems by reducing the positional tracking error by up to 82% when compared to state-of-the-art MPC approaches without neural network dynamics.
△ Less
Submitted 25 July, 2023; v1 submitted 15 March, 2022;
originally announced March 2022.
-
Motron: Multimodal Probabilistic Human Motion Forecasting
Authors:
Tim Salzmann,
Marco Pavone,
Markus Ryll
Abstract:
Autonomous systems and humans are increasingly sharing the same space. Robots work side by side or even hand in hand with humans to balance each other's limitations. Such cooperative interactions are ever more sophisticated. Thus, the ability to reason not just about a human's center of gravity position, but also its granular motion is an important prerequisite for human-robot interaction. Though,…
▽ More
Autonomous systems and humans are increasingly sharing the same space. Robots work side by side or even hand in hand with humans to balance each other's limitations. Such cooperative interactions are ever more sophisticated. Thus, the ability to reason not just about a human's center of gravity position, but also its granular motion is an important prerequisite for human-robot interaction. Though, many algorithms ignore the multimodal nature of humans or neglect uncertainty in their motion forecasts. We present Motron, a multimodal, probabilistic, graph-structured model, that captures human's multimodality using probabilistic methods while being able to output deterministic maximum-likelihood motions and corresponding confidence values for each mode. Our model aims to be tightly integrated with the robotic planning-control-interaction loop; outputting physically feasible human motions and being computationally efficient. We demonstrate the performance of our model on several challenging real-world motion forecasting datasets, outperforming a wide array of generative/variational methods while providing state-of-the-art single-output motions if required. Both using significantly less computational power than state-of-the art algorithms.
△ Less
Submitted 25 March, 2022; v1 submitted 8 March, 2022;
originally announced March 2022.
-
A caster-wheel-aware MPC-based motion planner for mobile robotics
Authors:
Jon Arrizabalaga,
Niels van Duijkeren,
Markus Ryll,
Ralph Lange
Abstract:
Differential drive mobile robots often use one or more caster wheels for balance. Caster wheels are appreciated for their ability to turn in any direction almost on the spot, allowing the robot to do the same and thereby greatly simplifying the motion planning and control. However, in aligning the caster wheels to the intended direction of motion they produce a so-called bore torque. As a result,…
▽ More
Differential drive mobile robots often use one or more caster wheels for balance. Caster wheels are appreciated for their ability to turn in any direction almost on the spot, allowing the robot to do the same and thereby greatly simplifying the motion planning and control. However, in aligning the caster wheels to the intended direction of motion they produce a so-called bore torque. As a result, additional motor torque is required to move the robot, which may in some cases exceed the motor capacity or compromise the motion planner's accuracy. Instead of taking a decoupled approach, where the navigation and disturbance rejection algorithms are separated, we propose to embed the caster wheel awareness into the motion planner. To do so, we present a caster-wheel-aware term that is compatible with MPC-based control methods, leveraging the existence of caster wheels in the motion planning stage. As a proof of concept, this term is combined with a a model-predictive trajectory tracking controller. Since this method requires knowledge of the caster wheel angle and rolling speed, an observer that estimates these states is also presented. The efficacy of the approach is shown in experiments on an intralogistics robot and compared against a decoupled bore-torque reduction approach and a caster-wheel agnostic controller. Moreover, the experiments show that the presented caster wheel estimator performs sufficiently well and therefore avoids the need for additional sensors.
△ Less
Submitted 11 October, 2021;
originally announced October 2021.
-
Towards Time-Optimal Tunnel-Following for Quadrotors
Authors:
Jon Arrizabalaga,
Markus Ryll
Abstract:
Minimum-time navigation within constrained and dynamic environments is of special relevance in robotics. Seeking time-optimality, while guaranteeing the integrity of time-varying spatial bounds, is an appealing trade-off for agile vehicles, such as quadrotors. State of the art approaches, either assume bounds to be static and generate time-optimal trajectories offline, or compromise time-optimalit…
▽ More
Minimum-time navigation within constrained and dynamic environments is of special relevance in robotics. Seeking time-optimality, while guaranteeing the integrity of time-varying spatial bounds, is an appealing trade-off for agile vehicles, such as quadrotors. State of the art approaches, either assume bounds to be static and generate time-optimal trajectories offline, or compromise time-optimality for constraint satisfaction. Leveraging nonlinear model predictive control and a path parametric reformulation of the quadrotor model, we present a real-time control that approximates time-optimal behavior and remains within dynamic corridors. The efficacy of the approach is evaluated according to simulated results, showing itself capable of performing extremely aggressive maneuvers as well as stop-and-go and backward motions.
△ Less
Submitted 3 June, 2022; v1 submitted 4 October, 2021;
originally announced October 2021.
-
FAST-Hex -- A Morphing Hexarotor: Design, Mechanical Implementation, Control and Experimental Validation
Authors:
Markus Ryll,
Davide Bicego,
Mattia Giurato,
Marco Lovera,
Antonio Franchi
Abstract:
We present FAST-Hex, a micro aerial hexarotor platform that allows to seamlessly transit from an under-actuated to a fully-actuated configuration with only one additional control input, a motor that synchronously tilts all propellers. The FAST-Hex adapts its configuration between the more efficient but under-actuated, collinear multi-rotors and the less efficient, but full-pose-tracking, which is…
▽ More
We present FAST-Hex, a micro aerial hexarotor platform that allows to seamlessly transit from an under-actuated to a fully-actuated configuration with only one additional control input, a motor that synchronously tilts all propellers. The FAST-Hex adapts its configuration between the more efficient but under-actuated, collinear multi-rotors and the less efficient, but full-pose-tracking, which is attained by non-collinear multi-rotors. On the basis of prior work on minimal input configurable micro aerial vehicle we mainly stress three aspects: mechanical design, motion control and experimental validation. Specifically, we present the lightweight mechanical structure of the FAST-Hex that allows it to only use one additional input to achieve configurability and full actuation in a vast state space. The motion controller receives as input any reference pose in $\mathbb{R}^3\times \mathrm{SO}(3)$ (3D position + 3D orientation). Full pose tracking is achieved if the reference pose is feasible with respect to actuator constraints. In case of unfeasibility a new feasible desired trajectory is generated online giving priority to the position tracking over the orientation tracking. Finally we present a large set of experimental results shading light on all aspects of the control and pose tracking of FAST-Hex.
△ Less
Submitted 14 April, 2020;
originally announced April 2020.
-
Full-Pose Tracking Control for Aerial Robotic Systems with Laterally-Bounded Input Force
Authors:
Antonio Franchi,
Ruggero Carli,
Davide Bicego,
Markus Ryll
Abstract:
In this paper, we define a general class of abstract aerial robotic systems named Laterally Bounded Force (LBF) vehicles, in which most of the control authority is expressed along a principal thrust direction, while in the lateral directions a (smaller and possibly null) force may be exploited to achieve full-pose tracking. This class approximates well platforms endowed with non-coplanar/non-colli…
▽ More
In this paper, we define a general class of abstract aerial robotic systems named Laterally Bounded Force (LBF) vehicles, in which most of the control authority is expressed along a principal thrust direction, while in the lateral directions a (smaller and possibly null) force may be exploited to achieve full-pose tracking. This class approximates well platforms endowed with non-coplanar/non-collinear rotors that can use the tilted propellers to slightly change the orientation of the total thrust w.r.t. the body frame. For this broad class of systems, we introduce a new geometric control strategy in SE(3) to achieve, whenever made possible by the force constraints, the independent tracking of position-plus-orientation trajectories. The exponential tracking of a feasible full-pose reference trajectory is proven using a Lyapunov technique in SE(3). The method can deal seamlessly with both under- and fully-actuated LBF platforms. The controller guarantees the tracking of at least the positional part in the case that an unfeasible full-pose reference trajectory is provided. The paper provides several experimental tests clearly showing the practicability of the approach and the sharp improvement with respect to state of-the-art approaches.
△ Less
Submitted 17 May, 2017; v1 submitted 21 May, 2016;
originally announced May 2016.