-
Crowdsourcing Task Traces for Service Robotics
Authors:
David Porfirio,
Allison Sauppé,
Maya Cakmak,
Aws Albarghouthi,
Bilge Mutlu
Abstract:
Demonstration is an effective end-user development paradigm for teaching robots how to perform new tasks. In this paper, we posit that demonstration is useful not only as a teaching tool, but also as a way to understand and assist end-user developers in thinking about a task at hand. As a first step toward gaining this understanding, we constructed a lightweight web interface to crowdsource step-b…
▽ More
Demonstration is an effective end-user development paradigm for teaching robots how to perform new tasks. In this paper, we posit that demonstration is useful not only as a teaching tool, but also as a way to understand and assist end-user developers in thinking about a task at hand. As a first step toward gaining this understanding, we constructed a lightweight web interface to crowdsource step-by-step instructions of common household tasks, leveraging the imaginations and past experiences of potential end-user developers. As evidence of the utility of our interface, we deployed the interface on Amazon Mechanical Turk and collected 207 task traces that span 18 different task categories. We describe our vision for how these task traces can be operationalized as task models within end-user development tools and provide a roadmap for future work.
△ Less
Submitted 20 March, 2024;
originally announced March 2024.
-
Multiple Ways of Working with Users to Develop Physically Assistive Robots
Authors:
Amal Nanavati,
Max Pascher,
Vinitha Ranganeni,
Ethan K. Gordon,
Taylor Kessler Faulkner,
Siddhartha S. Srinivasa,
Maya Cakmak,
Patrícia Alves-Oliveira,
Jens Gerken
Abstract:
Despite the growth of physically assistive robotics (PAR) research over the last decade, nearly half of PAR user studies do not involve participants with the target disabilities. There are several reasons for this -- recruitment challenges, small sample sizes, and transportation logistics -- all influenced by systemic barriers that people with disabilities face. However, it is well-established tha…
▽ More
Despite the growth of physically assistive robotics (PAR) research over the last decade, nearly half of PAR user studies do not involve participants with the target disabilities. There are several reasons for this -- recruitment challenges, small sample sizes, and transportation logistics -- all influenced by systemic barriers that people with disabilities face. However, it is well-established that working with end-users results in technology that better addresses their needs and integrates with their lived circumstances. In this paper, we reflect on multiple approaches we have taken to working with people with motor impairments across the design, development, and evaluation of three PAR projects: (a) assistive feeding with a robot arm; (b) assistive teleoperation with a mobile manipulator; and (c) shared control with a robot arm. We discuss these approaches to working with users along three dimensions -- individual vs. community-level insight, logistic burden on end-users vs. researchers, and benefit to researchers vs. community -- and share recommendations for how other PAR researchers can incorporate users into their work.
△ Less
Submitted 7 March, 2024; v1 submitted 1 March, 2024;
originally announced March 2024.
-
Fast Explicit-Input Assistance for Teleoperation in Clutter
Authors:
Nick Walker,
Xuning Yang,
Animesh Garg,
Maya Cakmak,
Dieter Fox,
Claudia Pérez-D'Arpino
Abstract:
The performance of prediction-based assistance for robot teleoperation degrades in unseen or goal-rich environments due to incorrect or quickly-changing intent inferences. Poor predictions can confuse operators or cause them to change their control input to implicitly signal their goal. We present a new assistance interface for robotic manipulation where an operator can explicitly communicate a ma…
▽ More
The performance of prediction-based assistance for robot teleoperation degrades in unseen or goal-rich environments due to incorrect or quickly-changing intent inferences. Poor predictions can confuse operators or cause them to change their control input to implicitly signal their goal. We present a new assistance interface for robotic manipulation where an operator can explicitly communicate a manipulation goal by pointing the end-effector. The pointing target specifies a region for local pose generation and optimization, providing interactive control over grasp and placement pose candidates. We compare the explicit pointing interface to an implicit inference-based assistance scheme in a within-subjects user study (N=20) where participants teleoperate a simulated robot to complete a multi-step singulation and stacking task in cluttered environments. We find that operators prefer the explicit interface, experience fewer pick failures and report lower cognitive workload. Our code is available at: https://github.com/NVlabs/fast-explicit-teleop
△ Less
Submitted 2 April, 2024; v1 submitted 4 February, 2024;
originally announced February 2024.
-
Evaluating Customization of Remote Tele-operation Interfaces for Assistive Robots
Authors:
Vinitha Ranganeni,
Noah Ponto,
Maya Cakmak
Abstract:
Mobile manipulator platforms, like the Stretch RE1 robot, make the promise of in-home robotic assistance feasible. For people with severe physical limitations, like those with quadriplegia, the ability to tele-operate these robots themselves means that they can perform physical tasks they cannot otherwise do themselves, thereby increasing their level of independence. In order for users with physic…
▽ More
Mobile manipulator platforms, like the Stretch RE1 robot, make the promise of in-home robotic assistance feasible. For people with severe physical limitations, like those with quadriplegia, the ability to tele-operate these robots themselves means that they can perform physical tasks they cannot otherwise do themselves, thereby increasing their level of independence. In order for users with physical limitations to operate these robots, their interfaces must be accessible and cater to the specific needs of all users. As physical limitations vary amongst users, it is difficult to make a single interface that will accommodate all users. Instead, such interfaces should be customizable to each individual user. In this paper we explore the value of customization of a browser-based interface for tele-operating the Stretch RE1 robot. More specifically, we evaluate the usability and effectiveness of a customized interface in comparison to the default interface configurations from prior work. We present a user study involving participants with motor impairments (N=10) and without motor impairments, who could serve as a caregiver, (N=13) that use the robot to perform mobile manipulation tasks in a real kitchen environment. Our study demonstrates that no single interface configuration satisfies all users' needs and preferences. Users perform better when using the customized interface for navigation, but not for manipulation due to higher complexity of learning to manipulate through the robot. All participants are able to use the robot to complete all tasks and participants with motor impairments believe that having the robot in their home would make them more independent.
△ Less
Submitted 5 April, 2023;
originally announced April 2023.
-
Sketching Robot Programs On the Fly
Authors:
David Porfirio,
Laura Stegner,
Maya Cakmak,
Allison Sauppé,
Aws Albarghouthi,
Bilge Mutlu
Abstract:
Service robots for personal use in the home and the workplace require end-user development solutions for swiftly scripting robot tasks as the need arises. Many existing solutions preserve ease, efficiency, and convenience through simple programming interfaces or by restricting task complexity. Others facilitate meticulous task design but often do so at the expense of simplicity and efficiency. The…
▽ More
Service robots for personal use in the home and the workplace require end-user development solutions for swiftly scripting robot tasks as the need arises. Many existing solutions preserve ease, efficiency, and convenience through simple programming interfaces or by restricting task complexity. Others facilitate meticulous task design but often do so at the expense of simplicity and efficiency. There is a need for robot programming solutions that reconcile the complexity of robotics with the on-the-fly goals of end-user development. In response to this need, we present a novel, multimodal, and on-the-fly development system, Tabula. Inspired by a formative design study with a prototype, Tabula leverages a combination of spoken language for specifying the core of a robot task and sketching for contextualizing the core. The result is that developers can script partial, sloppy versions of robot programs to be completed and refined by a program synthesizer. Lastly, we demonstrate our anticipated use cases of Tabula via a set of application scenarios.
△ Less
Submitted 6 February, 2023;
originally announced February 2023.
-
Participatory Design for Mental Health Data Visualization on a Social Robot
Authors:
Raida Karim,
Edgar Lopez,
Elin A. Björling,
Maya Cakmak
Abstract:
The intersection of data visualization and human-robot interaction (HRI) is a burgeoning field. Understanding, communicating, and processing different kinds of data for creating versatile visualizations can benefit HRI. Conversely, expressing different kinds of data generated from HRI through effective visualizations can provide interesting insights. Our work adds to the literature of this growing…
▽ More
The intersection of data visualization and human-robot interaction (HRI) is a burgeoning field. Understanding, communicating, and processing different kinds of data for creating versatile visualizations can benefit HRI. Conversely, expressing different kinds of data generated from HRI through effective visualizations can provide interesting insights. Our work adds to the literature of this growing domain. In this paper, we present our exploratory work on visualizing mental health data on a social robot. Particularly, we discuss development of mental health data visualizations using a participatory design (PD) approach. As a first step with mental health data visualization on a social robot, this work paves the way for relevant further work and using social robots as data visualization tools.
△ Less
Submitted 20 August, 2022;
originally announced October 2022.
-
Share with Me: A Study on a Social Robot Collecting Mental Health Data
Authors:
Raida Karim,
Edgar Lopez,
Katelynn Oleson,
Tony Li,
Elin A. Björling,
Maya Cakmak
Abstract:
Social robots have been used to assist with mental well-being in various ways such as to help children with autism improve on their social skills and executive functioning such as joint attention and bodily awareness. They are also used to help older adults by reducing feelings of isolation and loneliness, as well as supporting mental well-being of teens and children. However, existing work in thi…
▽ More
Social robots have been used to assist with mental well-being in various ways such as to help children with autism improve on their social skills and executive functioning such as joint attention and bodily awareness. They are also used to help older adults by reducing feelings of isolation and loneliness, as well as supporting mental well-being of teens and children. However, existing work in this sphere has only shown support for mental health through social robots by responding interactively to human activity to help them learn relevant skills. We hypothesize that humans can also get help from social robots in mental well-being by releasing or sharing their mental health data with the social robots. In this paper, we present a human-robot interaction (HRI) study to evaluate this hypothesis. During the five-day study, a total of fifty-five (n=55) participants shared their in-the-moment mood and stress levels with a social robot. We saw a majority of positive results indicating it is worth conducting future work in this direction, and the potential of social robots to largely support mental well-being.
△ Less
Submitted 8 August, 2022;
originally announced August 2022.
-
HandoverSim: A Simulation Framework and Benchmark for Human-to-Robot Object Handovers
Authors:
Yu-Wei Chao,
Chris Paxton,
Yu Xiang,
Wei Yang,
Balakumar Sundaralingam,
Tao Chen,
Adithyavairavan Murali,
Maya Cakmak,
Dieter Fox
Abstract:
We introduce a new simulation benchmark "HandoverSim" for human-to-robot object handovers. To simulate the giver's motion, we leverage a recent motion capture dataset of hand grasping of objects. We create training and evaluation environments for the receiver with standardized protocols and metrics. We analyze the performance of a set of baselines and show a correlation with a real-world evaluatio…
▽ More
We introduce a new simulation benchmark "HandoverSim" for human-to-robot object handovers. To simulate the giver's motion, we leverage a recent motion capture dataset of hand grasping of objects. We create training and evaluation environments for the receiver with standardized protocols and metrics. We analyze the performance of a set of baselines and show a correlation with a real-world evaluation. Code is open sourced at https://handover-sim.github.io.
△ Less
Submitted 19 May, 2022;
originally announced May 2022.
-
Model Predictive Control for Fluid Human-to-Robot Handovers
Authors:
Wei Yang,
Balakumar Sundaralingam,
Chris Paxton,
Iretiayo Akinola,
Yu-Wei Chao,
Maya Cakmak,
Dieter Fox
Abstract:
Human-robot handover is a fundamental yet challenging task in human-robot interaction and collaboration. Recently, remarkable progressions have been made in human-to-robot handovers of unknown objects by using learning-based grasp generators. However, how to responsively generate smooth motions to take an object from a human is still an open question. Specifically, planning motions that take human…
▽ More
Human-robot handover is a fundamental yet challenging task in human-robot interaction and collaboration. Recently, remarkable progressions have been made in human-to-robot handovers of unknown objects by using learning-based grasp generators. However, how to responsively generate smooth motions to take an object from a human is still an open question. Specifically, planning motions that take human comfort into account is not a part of the human-robot handover process in most prior works. In this paper, we propose to generate smooth motions via an efficient model-predictive control (MPC) framework that integrates perception and complex domain-specific constraints into the optimization problem. We introduce a learning-based grasp reachability model to select candidate grasps which maximize the robot's manipulability, giving it more freedom to satisfy these constraints. Finally, we integrate a neural net force/torque classifier that detects contact events from noisy data. We conducted human-to-robot handover experiments on a diverse set of objects with several users (N=4) and performed a systematic evaluation of each module. The study shows that the users preferred our MPC approach over the baseline system by a large margin. More results and videos are available at https://sites.google.com/nvidia.com/mpc-for-handover.
△ Less
Submitted 31 March, 2022;
originally announced April 2022.
-
Assistive Tele-op: Leveraging Transformers to Collect Robotic Task Demonstrations
Authors:
Henry M. Clever,
Ankur Handa,
Hammad Mazhar,
Kevin Parker,
Omer Shapira,
Qian Wan,
Yashraj Narang,
Iretiayo Akinola,
Maya Cakmak,
Dieter Fox
Abstract:
Sharing autonomy between robots and human operators could facilitate data collection of robotic task demonstrations to continuously improve learned models. Yet, the means to communicate intent and reason about the future are disparate between humans and robots. We present Assistive Tele-op, a virtual reality (VR) system for collecting robot task demonstrations that displays an autonomous trajector…
▽ More
Sharing autonomy between robots and human operators could facilitate data collection of robotic task demonstrations to continuously improve learned models. Yet, the means to communicate intent and reason about the future are disparate between humans and robots. We present Assistive Tele-op, a virtual reality (VR) system for collecting robot task demonstrations that displays an autonomous trajectory forecast to communicate the robot's intent. As the robot moves, the user can switch between autonomous and manual control when desired. This allows users to collect task demonstrations with both a high success rate and with greater ease than manual teleoperation systems. Our system is powered by transformers, which can provide a window of potential states and actions far into the future -- with almost no added computation time. A key insight is that human intent can be injected at any location within the transformer sequence if the user decides that the model-predicted actions are inappropriate. At every time step, the user can (1) do nothing and allow autonomous operation to continue while observing the robot's future plan sequence, or (2) take over and momentarily prescribe a different set of actions to nudge the model back on track. We host the videos and other supplementary material at https://sites.google.com/view/assistive-teleop.
△ Less
Submitted 9 December, 2021;
originally announced December 2021.
-
Learning Perceptual Concepts by Bootstrapping from Human Queries
Authors:
Andreea Bobu,
Chris Paxton,
Wei Yang,
Balakumar Sundaralingam,
Yu-Wei Chao,
Maya Cakmak,
Dieter Fox
Abstract:
When robots operate in human environments, it's critical that humans can quickly teach them new concepts: object-centric properties of the environment that they care about (e.g. objects near, upright, etc). However, teaching a new perceptual concept from high-dimensional robot sensor data (e.g. point clouds) is demanding, requiring an unrealistic amount of human labels. To address this, we propose…
▽ More
When robots operate in human environments, it's critical that humans can quickly teach them new concepts: object-centric properties of the environment that they care about (e.g. objects near, upright, etc). However, teaching a new perceptual concept from high-dimensional robot sensor data (e.g. point clouds) is demanding, requiring an unrealistic amount of human labels. To address this, we propose a framework called Perceptual Concept Bootstrapping (PCB). First, we leverage the inherently lower-dimensional privileged information, e.g., object poses and bounding boxes, available from a simulator only at training time to rapidly learn a low-dimensional, geometric concept from minimal human input. Second, we treat this low-dimensional concept as an automatic labeler to synthesize a large-scale high-dimensional data set with the simulator. With these two key ideas, PCB alleviates human label burden while still learning perceptual concepts that work with real sensor input where no privileged information is available. We evaluate PCB for learning spatial concepts that describe object state or multi-object relationships, and show it achieves superior performance compared to baseline methods. We also demonstrate the utility of the learned concepts in motion planning tasks on a 7-DoF Franka Panda robot.
△ Less
Submitted 4 July, 2022; v1 submitted 9 November, 2021;
originally announced November 2021.
-
Reactive Human-to-Robot Handovers of Arbitrary Objects
Authors:
Wei Yang,
Chris Paxton,
Arsalan Mousavian,
Yu-Wei Chao,
Maya Cakmak,
Dieter Fox
Abstract:
Human-robot object handovers have been an actively studied area of robotics over the past decade; however, very few techniques and systems have addressed the challenge of handing over diverse objects with arbitrary appearance, size, shape, and rigidity. In this paper, we present a vision-based system that enables reactive human-to-robot handovers of unknown objects. Our approach combines closed-lo…
▽ More
Human-robot object handovers have been an actively studied area of robotics over the past decade; however, very few techniques and systems have addressed the challenge of handing over diverse objects with arbitrary appearance, size, shape, and rigidity. In this paper, we present a vision-based system that enables reactive human-to-robot handovers of unknown objects. Our approach combines closed-loop motion planning with real-time, temporally-consistent grasp generation to ensure reactivity and motion smoothness. Our system is robust to different object positions and orientations, and can grasp both rigid and non-rigid objects. We demonstrate the generalizability, usability, and robustness of our approach on a novel benchmark set of 26 diverse household objects, a user study with naive users (N=6) handing over a subset of 15 objects, and a systematic evaluation examining different ways of handing objects. More results and videos can be found at https://sites.google.com/nvidia.com/handovers-of-arbitrary-objects.
△ Less
Submitted 3 June, 2021; v1 submitted 17 November, 2020;
originally announced November 2020.
-
Affordance-Aware Handovers with Human Arm Mobility Constraints
Authors:
Paola Ardón,
Maria E. Cabrera,
Èric Pairet,
Ronald P. A. Petrick,
Subramanian Ramamoorthy,
Katrin S. Lohan,
Maya Cakmak
Abstract:
Reasoning about object handover configurations allows an assistive agent to estimate the appropriateness of handover for a receiver with different arm mobility capacities. While there are existing approaches for estimating the effectiveness of handovers, their findings are limited to users without arm mobility impairments and to specific objects. Therefore, current state-of-the-art approaches are…
▽ More
Reasoning about object handover configurations allows an assistive agent to estimate the appropriateness of handover for a receiver with different arm mobility capacities. While there are existing approaches for estimating the effectiveness of handovers, their findings are limited to users without arm mobility impairments and to specific objects. Therefore, current state-of-the-art approaches are unable to hand over novel objects to receivers with different arm mobility capacities. We propose a method that generalises handover behaviours to previously unseen objects, subject to the constraint of a user's arm mobility levels and the task context. We propose a heuristic-guided hierarchically optimised cost whose optimisation adapts object configurations for receivers with low arm mobility. This also ensures that the robot grasps consider the context of the user's upcoming task, i.e., the usage of the object. To understand preferences over handover configurations, we report on the findings of an online study, wherein we presented different handover methods, including ours, to $259$ users with different levels of arm mobility. We find that people's preferences over handover methods are correlated to their arm mobility capacities. We encapsulate these preferences in a statistical relational model (SRL) that is able to reason about the most suitable handover configuration given a receiver's arm mobility and upcoming task. Using our SRL model, we obtained an average handover accuracy of $90.8\%$ when generalising handovers to novel objects.
△ Less
Submitted 16 February, 2021; v1 submitted 29 October, 2020;
originally announced October 2020.
-
Human Grasp Classification for Reactive Human-to-Robot Handovers
Authors:
Wei Yang,
Chris Paxton,
Maya Cakmak,
Dieter Fox
Abstract:
Transfer of objects between humans and robots is a critical capability for collaborative robots. Although there has been a recent surge of interest in human-robot handovers, most prior research focus on robot-to-human handovers. Further, work on the equally critical human-to-robot handovers often assumes humans can place the object in the robot's gripper. In this paper, we propose an approach for…
▽ More
Transfer of objects between humans and robots is a critical capability for collaborative robots. Although there has been a recent surge of interest in human-robot handovers, most prior research focus on robot-to-human handovers. Further, work on the equally critical human-to-robot handovers often assumes humans can place the object in the robot's gripper. In this paper, we propose an approach for human-to-robot handovers in which the robot meets the human halfway, by classifying the human's grasp of the object and quickly planning a trajectory accordingly to take the object from the human's hand according to their intent. To do this, we collect a human grasp dataset which covers typical ways of holding objects with various hand shapes and poses, and learn a deep model on this dataset to classify the hand grasps into one of these categories. We present a planning and execution approach that takes the object from the human hand according to the detected grasp and hand position, and replans as necessary when the handover is interrupted. Through a systematic evaluation, we demonstrate that our system results in more fluent handovers versus two baselines. We also present findings from a user study (N = 9) demonstrating the effectiveness and usability of our approach with naive users in different scenarios. More results and videos can be found at http://wyang.me/handovers.
△ Less
Submitted 12 March, 2020;
originally announced March 2020.
-
Vision-and-Dialog Navigation
Authors:
Jesse Thomason,
Michael Murray,
Maya Cakmak,
Luke Zettlemoyer
Abstract:
Robots navigating in human environments should use language to ask for assistance and be able to understand human responses. To study this challenge, we introduce Cooperative Vision-and-Dialog Navigation, a dataset of over 2k embodied, human-human dialogs situated in simulated, photorealistic home environments. The Navigator asks questions to their partner, the Oracle, who has privileged access to…
▽ More
Robots navigating in human environments should use language to ask for assistance and be able to understand human responses. To study this challenge, we introduce Cooperative Vision-and-Dialog Navigation, a dataset of over 2k embodied, human-human dialogs situated in simulated, photorealistic home environments. The Navigator asks questions to their partner, the Oracle, who has privileged access to the best next steps the Navigator should take according to a shortest path planner. To train agents that search an environment for a goal location, we define the Navigation from Dialog History task. An agent, given a target object and a dialog history between humans cooperating to find that object, must infer navigation actions towards the goal in unexplored environments. We establish an initial, multi-modal sequence-to-sequence model and demonstrate that looking farther back in the dialog history improves performance. Sourcecode and a live interface demo can be found at https://cvdn.dev/
△ Less
Submitted 12 October, 2019; v1 submitted 10 July, 2019;
originally announced July 2019.
-
Desiderata for Planning Systems in General-Purpose Service Robots
Authors:
Nick Walker,
Yuqian Jiang,
Maya Cakmak,
Peter Stone
Abstract:
General-purpose service robots are expected to undertake a broad range of tasks at the request of users. Knowledge representation and planning systems are essential to flexible autonomous robots, but the field lacks a unified perspective on which features are essential for general-purpose service robots. Progress towards planning and reasoning for general-purpose service robots is hindered by diff…
▽ More
General-purpose service robots are expected to undertake a broad range of tasks at the request of users. Knowledge representation and planning systems are essential to flexible autonomous robots, but the field lacks a unified perspective on which features are essential for general-purpose service robots. Progress towards planning and reasoning for general-purpose service robots is hindered by differing assumptions about users, the environment, and the overall robot system. In this position paper, we propose desiderata for planning and reasoning systems to promote general-purpose service robots. Each proposed item draws on our experience with research on service robots in the office and home and on the demands of these environments. Our desiderata emphasize support for natural human-interfaces as well as for robust fallback methods when interactions with humans and the environment fail. We highlight relevant work towards these goals.
△ Less
Submitted 4 July, 2019;
originally announced July 2019.
-
Neural Semantic Parsing with Anonymization for Command Understanding in General-Purpose Service Robots
Authors:
Nick Walker,
Yu-Tang Peng,
Maya Cakmak
Abstract:
Service robots are envisioned to undertake a wide range of tasks at the request of users. Semantic parsing is one way to convert natural language commands given to these robots into executable representations. Methods for creating semantic parsers, however, rely either on large amounts of data or on engineered lexical features and parsing rules, which has limited their application in robotics. To…
▽ More
Service robots are envisioned to undertake a wide range of tasks at the request of users. Semantic parsing is one way to convert natural language commands given to these robots into executable representations. Methods for creating semantic parsers, however, rely either on large amounts of data or on engineered lexical features and parsing rules, which has limited their application in robotics. To address this challenge, we propose an approach that leverages neural semantic parsing methods in combination with contextual word embeddings to enable the training of a semantic parser with little data and without domain specific parser engineering. Key to our approach is the use of an anonymized target representation which is more easily learned by the parser. In most cases, this simplified representation can trivially be transformed into an executable format, and in others the parse can be completed through further interaction with the user. We evaluate this approach in the context of the RoboCup@Home General Purpose Service Robot task, where we have collected a corpus of paraphrased versions of commands from the standardized command generator. Our results show that neural semantic parsers can predict the logical form of unseen commands with 89% accuracy. We release our data and the details of our models to encourage further development from the RoboCup and service robotics communities.
△ Less
Submitted 1 July, 2019;
originally announced July 2019.
-
Programming by Demonstration with User-Specified Perceptual Landmarks
Authors:
Justin Huang,
Maya Cakmak
Abstract:
Programming by demonstration (PbD) is an effective technique for developing complex robot manipulation tasks, such as opening bottles or using human tools. In order for such tasks to generalize to new scenes, the robot needs to be able to perceive objects, object parts, or other task-relevant parts of the scene. Previous work has relied on rigid, task-specific perception systems for this purpose.…
▽ More
Programming by demonstration (PbD) is an effective technique for developing complex robot manipulation tasks, such as opening bottles or using human tools. In order for such tasks to generalize to new scenes, the robot needs to be able to perceive objects, object parts, or other task-relevant parts of the scene. Previous work has relied on rigid, task-specific perception systems for this purpose. This paper presents a flexible and open-ended perception system that lets users specify perceptual "landmarks" during the demonstration, by capturing parts of the point cloud from the demonstration scene. We present a method for localizing landmarks in new scenes and experimentally evaluate this method in a variety of settings. Then, we provide examples where user-specified landmarks are used together with PbD on a PR2 robot to perform several complex manipulation tasks. Finally, we present findings from a user evaluation of our landmark specification interface demonstrating its feasibility as an end-user tool.
△ Less
Submitted 1 December, 2016;
originally announced December 2016.
