-
MAVERICK: An App-independent and Platform-agnostic Approach to Enforce Policies in IoT Systems at Runtime
Authors:
M. Hammad Mazhar,
Li Li,
Endadul Hoque,
Omar Chowdhury
Abstract:
Many solutions have been proposed to curb unexpected behavior of automation apps installed on programmable IoT platforms by enforcing safety policies at runtime. However, all prior work addresses a weaker version of the actual problem due to a simpler, unrealistic threat model. These solutions are not general enough as they are heavily dependent on the installed apps and catered to specific IoT pl…
▽ More
Many solutions have been proposed to curb unexpected behavior of automation apps installed on programmable IoT platforms by enforcing safety policies at runtime. However, all prior work addresses a weaker version of the actual problem due to a simpler, unrealistic threat model. These solutions are not general enough as they are heavily dependent on the installed apps and catered to specific IoT platforms. Here, we address a stronger version of the problem via a realistic threat model, where (i) undesired cyber actions can come from not only automation platform backends (e.g., SmartThings) but also close-sourced third-party services (e.g., IFTTT), and (ii) physical actions (e.g., user interactions) on devices can move the IoT system to an undesirable state. We propose a runtime mechanism, dubbed Maverick, which employs an app-independent, platform-agnostic mediator to enforce policies against all undesired cyber actions and applies corrective-actions to bring the IoT system back to a safe state from an unsafe state transition. Maverick is equipped with a policy language capable of expressing rich temporal invariants and an automated toolchain that includes a policy synthesizer and a policy analyzer for user assistance. We implemented Maverick in a prototype and showed its efficacy in both physical and virtual testbeds, incurring minimal overhead.
△ Less
Submitted 18 April, 2023; v1 submitted 2 February, 2023;
originally announced February 2023.
-
Orbit: A Unified Simulation Framework for Interactive Robot Learning Environments
Authors:
Mayank Mittal,
Calvin Yu,
Qinxi Yu,
Jingzhou Liu,
Nikita Rudin,
David Hoeller,
Jia Lin Yuan,
Ritvik Singh,
Yunrong Guo,
Hammad Mazhar,
Ajay Mandlekar,
Buck Babich,
Gavriel State,
Marco Hutter,
Animesh Garg
Abstract:
We present Orbit, a unified and modular framework for robot learning powered by NVIDIA Isaac Sim. It offers a modular design to easily and efficiently create robotic environments with photo-realistic scenes and high-fidelity rigid and deformable body simulation. With Orbit, we provide a suite of benchmark tasks of varying difficulty -- from single-stage cabinet opening and cloth folding to multi-s…
▽ More
We present Orbit, a unified and modular framework for robot learning powered by NVIDIA Isaac Sim. It offers a modular design to easily and efficiently create robotic environments with photo-realistic scenes and high-fidelity rigid and deformable body simulation. With Orbit, we provide a suite of benchmark tasks of varying difficulty -- from single-stage cabinet opening and cloth folding to multi-stage tasks such as room reorganization. To support working with diverse observations and action spaces, we include fixed-arm and mobile manipulators with different physically-based sensors and motion generators. Orbit allows training reinforcement learning policies and collecting large demonstration datasets from hand-crafted or expert solutions in a matter of minutes by leveraging GPU-based parallelization. In summary, we offer an open-sourced framework that readily comes with 16 robotic platforms, 4 sensor modalities, 10 motion generators, more than 20 benchmark tasks, and wrappers to 4 learning libraries. With this framework, we aim to support various research areas, including representation learning, reinforcement learning, imitation learning, and task and motion planning. We hope it helps establish interdisciplinary collaborations in these communities, and its modularity makes it easily extensible for more tasks and applications in the future.
△ Less
Submitted 16 February, 2024; v1 submitted 10 January, 2023;
originally announced January 2023.
-
Assistive Tele-op: Leveraging Transformers to Collect Robotic Task Demonstrations
Authors:
Henry M. Clever,
Ankur Handa,
Hammad Mazhar,
Kevin Parker,
Omer Shapira,
Qian Wan,
Yashraj Narang,
Iretiayo Akinola,
Maya Cakmak,
Dieter Fox
Abstract:
Sharing autonomy between robots and human operators could facilitate data collection of robotic task demonstrations to continuously improve learned models. Yet, the means to communicate intent and reason about the future are disparate between humans and robots. We present Assistive Tele-op, a virtual reality (VR) system for collecting robot task demonstrations that displays an autonomous trajector…
▽ More
Sharing autonomy between robots and human operators could facilitate data collection of robotic task demonstrations to continuously improve learned models. Yet, the means to communicate intent and reason about the future are disparate between humans and robots. We present Assistive Tele-op, a virtual reality (VR) system for collecting robot task demonstrations that displays an autonomous trajectory forecast to communicate the robot's intent. As the robot moves, the user can switch between autonomous and manual control when desired. This allows users to collect task demonstrations with both a high success rate and with greater ease than manual teleoperation systems. Our system is powered by transformers, which can provide a window of potential states and actions far into the future -- with almost no added computation time. A key insight is that human intent can be injected at any location within the transformer sequence if the user decides that the model-predicted actions are inappropriate. At every time step, the user can (1) do nothing and allow autonomous operation to continue while observing the robot's future plan sequence, or (2) take over and momentarily prescribe a different set of actions to nudge the model back on track. We host the videos and other supplementary material at https://sites.google.com/view/assistive-teleop.
△ Less
Submitted 9 December, 2021;
originally announced December 2021.
-
RGB-D Local Implicit Function for Depth Completion of Transparent Objects
Authors:
Luyang Zhu,
Arsalan Mousavian,
Yu Xiang,
Hammad Mazhar,
Jozef van Eenbergen,
Shoubhik Debnath,
Dieter Fox
Abstract:
Majority of the perception methods in robotics require depth information provided by RGB-D cameras. However, standard 3D sensors fail to capture depth of transparent objects due to refraction and absorption of light. In this paper, we introduce a new approach for depth completion of transparent objects from a single RGB-D image. Key to our approach is a local implicit neural representation built o…
▽ More
Majority of the perception methods in robotics require depth information provided by RGB-D cameras. However, standard 3D sensors fail to capture depth of transparent objects due to refraction and absorption of light. In this paper, we introduce a new approach for depth completion of transparent objects from a single RGB-D image. Key to our approach is a local implicit neural representation built on ray-voxel pairs that allows our method to generalize to unseen objects and achieve fast inference speed. Based on this representation, we present a novel framework that can complete missing depth given noisy RGB-D input. We further improve the depth estimation iteratively using a self-correcting refinement model. To train the whole pipeline, we build a large scale synthetic dataset with transparent objects. Experiments demonstrate that our method performs significantly better than the current state-of-the-art methods on both synthetic and real world data. In addition, our approach improves the inference speed by a factor of 20 compared to the previous best method, ClearGrasp. Code and dataset will be released at https://research.nvidia.com/publication/2021-03_RGB-D-Local-Implicit.
△ Less
Submitted 1 April, 2021;
originally announced April 2021.
-
Transferable Task Execution from Pixels through Deep Planning Domain Learning
Authors:
Kei Kase,
Chris Paxton,
Hammad Mazhar,
Tetsuya Ogata,
Dieter Fox
Abstract:
While robots can learn models to solve many manipulation tasks from raw visual input, they cannot usually use these models to solve new problems. On the other hand, symbolic planning methods such as STRIPS have long been able to solve new problems given only a domain definition and a symbolic goal, but these approaches often struggle on the real world robotic tasks due to the challenges of groundi…
▽ More
While robots can learn models to solve many manipulation tasks from raw visual input, they cannot usually use these models to solve new problems. On the other hand, symbolic planning methods such as STRIPS have long been able to solve new problems given only a domain definition and a symbolic goal, but these approaches often struggle on the real world robotic tasks due to the challenges of grounding these symbols from sensor data in a partially-observable world. We propose Deep Planning Domain Learning (DPDL), an approach that combines the strengths of both methods to learn a hierarchical model. DPDL learns a high-level model which predicts values for a large set of logical predicates consisting of the current symbolic world state, and separately learns a low-level policy which translates symbolic operators into executable actions on the robot. This allows us to perform complex, multi-step tasks even when the robot has not been explicitly trained on them. We show our method on manipulation tasks in a photorealistic kitchen scenario.
△ Less
Submitted 8 March, 2020;
originally announced March 2020.
-
Characterizing Smart Home IoT Traffic in the Wild
Authors:
M. Hammad Mazhar,
Zubair Shafiq
Abstract:
As the smart home IoT ecosystem flourishes, it is imperative to gain a better understanding of the unique challenges it poses in terms of management, security, and privacy. Prior studies are limited because they examine smart home IoT devices in testbed environments or at a small scale. To address this gap, we present a measurement study of smart home IoT devices in the wild by instrumenting home…
▽ More
As the smart home IoT ecosystem flourishes, it is imperative to gain a better understanding of the unique challenges it poses in terms of management, security, and privacy. Prior studies are limited because they examine smart home IoT devices in testbed environments or at a small scale. To address this gap, we present a measurement study of smart home IoT devices in the wild by instrumenting home gateways and passively collecting real-world network traffic logs from more than 200 homes across a large metropolitan area in the United States. We characterize smart home IoT traffic in terms of its volume, temporal patterns, and external endpoints along with focusing on certain security and privacy concerns. We first show that traffic characteristics reflect the functionality of smart home IoT devices such as smart TVs generating high volume traffic to content streaming services following diurnal patterns associated with human activity. While the smart home IoT ecosystem seems fragmented, our analysis reveals that it is mostly centralized due to its reliance on a few popular cloud and DNS services. Our findings also highlight several interesting security and privacy concerns in smart home IoT ecosystem such as the need to improve policy-based access control for IoT traffic, lack of use of application layer encryption, and prevalence of third-party advertising and tracking services. Our findings have important implications for future research on improving management, security, and privacy of the smart home IoT ecosystem.
△ Less
Submitted 30 March, 2020; v1 submitted 22 January, 2020;
originally announced January 2020.
