-
ToolFlowNet: Robotic Manipulation with Tools via Predicting Tool Flow from Point Clouds
Authors:
Daniel Seita,
Yufei Wang,
Sarthak J. Shetty,
Edward Yao Li,
Zackory Erickson,
David Held
Abstract:
Point clouds are a widely available and canonical data modality which convey the 3D geometry of a scene. Despite significant progress in classification and segmentation from point clouds, policy learning from such a modality remains challenging, and most prior works in imitation learning focus on learning policies from images or state information. In this paper, we propose a novel framework for le…
▽ More
Point clouds are a widely available and canonical data modality which convey the 3D geometry of a scene. Despite significant progress in classification and segmentation from point clouds, policy learning from such a modality remains challenging, and most prior works in imitation learning focus on learning policies from images or state information. In this paper, we propose a novel framework for learning policies from point clouds for robotic manipulation with tools. We use a novel neural network, ToolFlowNet, which predicts dense per-point flow on the tool that the robot controls, and then uses the flow to derive the transformation that the robot should execute. We apply this framework to imitation learning of challenging deformable object manipulation tasks with continuous movement of tools, including scooping and pouring, and demonstrate significantly improved performance over baselines which do not use flow. We perform 50 physical scooping experiments with ToolFlowNet and attain 82% scooping success. See https://tinyurl.com/toolflownet for supplementary material.
△ Less
Submitted 16 November, 2022;
originally announced November 2022.
-
Weight-Based Exploration for Unmanned Aerial Teams Searching for Multiple Survivors
Authors:
Sarthak J. Shetty,
Debasish Ghose
Abstract:
During floods, reaching survivors in the shortest possible time is a priority for rescue teams. Given their ability to explore difficult terrain in short spans of time, Unmanned Aerial Vehicles (UAVs) have become an increasingly valuable aid to search and rescue operations. Traditionally, UAVs utilize exhaustive lawnmower exploration patterns to locate stranded survivors, without any information r…
▽ More
During floods, reaching survivors in the shortest possible time is a priority for rescue teams. Given their ability to explore difficult terrain in short spans of time, Unmanned Aerial Vehicles (UAVs) have become an increasingly valuable aid to search and rescue operations. Traditionally, UAVs utilize exhaustive lawnmower exploration patterns to locate stranded survivors, without any information regarding the survivor's whereabouts. In real life disaster scenarios however, on-ground observers provide valuable information to the rescue effort, such as the survivor's last known location and heading. In earlier work, a Weight Based Exploration (WBE) model, which utilizes this information to generate a prioritized list of waypoints to aid the UAV in its search mission, was proposed. This approach was shown to be effective for a single UAV locating a single survivor. In this paper, we extend the WBE model to a team of UAVs locating multiple survivors. The model initially partitions the search environment amongst the UAVs using Voronoi cells. The UAVs then utilize the WBE model to locate survivors in their partitions. We test this model with varying survivor locations and headings. We demonstrate the scalability of the model developed by testing the model with aerial teams comprising several UAVs.
△ Less
Submitted 21 December, 2020;
originally announced December 2020.
-
Implementation of Survivor Detection Strategies Using Drones
Authors:
Sarthak J. Shetty,
Rahul Ravichandran,
Lima Agnel Tony,
N. Sai Abhinay,
Kaushik Das,
Debasish Ghose
Abstract:
Survivors stranded during floods tend to seek refuge on dry land. It is important to search for these survivors and help them reach safety as quickly as possible. The terrain in such situations however, is heavily damaged and restricts the movement of emergency personnel towards these survivors. Therefore, it is advantageous to utilize Unmanned Aerial Vehicles (UAVs) in cooperation with on-ground…
▽ More
Survivors stranded during floods tend to seek refuge on dry land. It is important to search for these survivors and help them reach safety as quickly as possible. The terrain in such situations however, is heavily damaged and restricts the movement of emergency personnel towards these survivors. Therefore, it is advantageous to utilize Unmanned Aerial Vehicles (UAVs) in cooperation with on-ground first responders to aid search and rescue efforts. In this article we demonstrate an implementation and improvement of the weight-based path planning algorithm using an off-the-shelf UAV. The coordinates of the survivor and their heading is reported by an on-ground observer to the UAV to generate a weighted map of the surroundings for exploration. Each coordinate in the map is assigned a weight which dictates the priority of exploration. These waypoints are then sorted on the basis of their weights to arrive at an ordered list for exploration by the UAV. We developed the model in MATLAB, followed by prototyping on Robot Operating System (ROS) using a 3DR Iris quadcopter. We tested the model on an off-the-shelf UAV by utilizing the MAVROS and MAVLINK capabilities of ROS. During the implementation of the algorithm on the UAV, several additional factors such as unreliable GPS signals and limited field of view which could effect the performance of the model were in effect, despite which the algorithm performed fairly well. We compared our model with conventional algorithms described in the literature, and showed that our implementation outperforms them.
△ Less
Submitted 4 April, 2020; v1 submitted 27 March, 2020;
originally announced March 2020.
-
Vision-based inspection system employing computer vision & neural networks for detection of fractures in manufactured components
Authors:
Sarthak J Shetty
Abstract:
We are proceeding towards the age of automation and robotic integration of our production lines [5]. Effective quality-control systems have to be put in place to maintain the quality of manufactured components. Among different quality-control systems, vision-based inspection systems have gained considerable amount of popularity [8] due to developments in computing power and image processing techni…
▽ More
We are proceeding towards the age of automation and robotic integration of our production lines [5]. Effective quality-control systems have to be put in place to maintain the quality of manufactured components. Among different quality-control systems, vision-based inspection systems have gained considerable amount of popularity [8] due to developments in computing power and image processing techniques. In this paper, we present a vision-based inspection system (VBI) as a quality-control system, which not only detects the presence of defects, such as in conventional VBIs, but also leverage developments in machine learning to predict the presence of surface fractures and wearing. We use OpenCV, an open source computer-vision framework, and Tensorflow, an open source machine-learning framework developed by Google Inc., to accomplish the tasks of detection and prediction of presence of surface defects such as fractures of manufactured gears.
△ Less
Submitted 25 January, 2019;
originally announced January 2019.
