-
Tactile SoftHand-A: 3D-Printed, Tactile, Highly-underactuated, Anthropomorphic Robot Hand with an Antagonistic Tendon Mechanism
Authors:
Haoran Li,
Christopher J. Ford,
Chenghua Lu,
Yijiong Lin,
Matteo Bianchi,
Manuel G. Catalano,
Efi Psomopoulou,
Nathan F. Lepora
Abstract:
For tendon-driven multi-fingered robotic hands, ensuring grasp adaptability while minimizing the number of actuators needed to provide human-like functionality is a challenging problem. Inspired by the Pisa/IIT SoftHand, this paper introduces a 3D-printed, highly-underactuated, five-finger robotic hand named the Tactile SoftHand-A, which features only two actuators. The dual-tendon design allows f…
▽ More
For tendon-driven multi-fingered robotic hands, ensuring grasp adaptability while minimizing the number of actuators needed to provide human-like functionality is a challenging problem. Inspired by the Pisa/IIT SoftHand, this paper introduces a 3D-printed, highly-underactuated, five-finger robotic hand named the Tactile SoftHand-A, which features only two actuators. The dual-tendon design allows for the active control of specific (distal or proximal interphalangeal) joints to adjust the hand's grasp gesture. We have also developed a new design of fully 3D-printed tactile sensor that requires no hand assembly and is printed directly as part of the robotic finger. This sensor is integrated into the fingertips and combined with the antagonistic tendon mechanism to develop a human-hand-guided tactile feedback grasping system. The system can actively mirror human hand gestures, adaptively stabilize grasp gestures upon contact, and adjust grasp gestures to prevent object movement after detecting slippage. Finally, we designed four different experiments to evaluate the novel fingers coupled with the antagonistic mechanism for controlling the robotic hand's gestures, adaptive grasping ability, and human-hand-guided tactile feedback grasping capability. The experimental results demonstrate that the Tactile SoftHand-A can adaptively grasp objects of a wide range of shapes and automatically adjust its gripping gestures upon detecting contact and slippage. Overall, this study points the way towards a class of low-cost, accessible, 3D-printable, underactuated human-like robotic hands, and we openly release the designs to facilitate others to build upon this work. This work is Open-sourced at github.com/SoutheastWind/Tactile_SoftHand_A
△ Less
Submitted 18 June, 2024;
originally announced June 2024.
-
AnyRotate: Gravity-Invariant In-Hand Object Rotation with Sim-to-Real Touch
Authors:
Max Yang,
Chenghua Lu,
Alex Church,
Yijiong Lin,
Chris Ford,
Haoran Li,
Efi Psomopoulou,
David A. W. Barton,
Nathan F. Lepora
Abstract:
Human hands are capable of in-hand manipulation in the presence of different hand motions. For a robot hand, harnessing rich tactile information to achieve this level of dexterity still remains a significant challenge. In this paper, we present AnyRotate, a system for gravity-invariant multi-axis in-hand object rotation using dense featured sim-to-real touch. We tackle this problem by training a d…
▽ More
Human hands are capable of in-hand manipulation in the presence of different hand motions. For a robot hand, harnessing rich tactile information to achieve this level of dexterity still remains a significant challenge. In this paper, we present AnyRotate, a system for gravity-invariant multi-axis in-hand object rotation using dense featured sim-to-real touch. We tackle this problem by training a dense tactile policy in simulation and present a sim-to-real method for rich tactile sensing to achieve zero-shot policy transfer. Our formulation allows the training of a unified policy to rotate unseen objects about arbitrary rotation axes in any hand direction. In our experiments, we highlight the benefit of capturing detailed contact information when handling objects with varying properties. Interestingly, despite not having explicit slip detection, we found rich multi-fingered tactile sensing can implicitly detect object movement within grasp and provide a reactive behavior that improves the robustness of the policy. The project website can be found at https://maxyang27896.github.io/anyrotate/.
△ Less
Submitted 11 June, 2024; v1 submitted 12 May, 2024;
originally announced May 2024.
-
DexiTac: Soft Dexterous Tactile Gripping
Authors:
Chenghua Lu,
Kailuan Tang,
Max Yang,
Tianqi Yue,
Nathan F. Lepora
Abstract:
Grasping object,whether they are flat, round, or narrow and whether they have regular or irregular shapes,introduces difficulties in determining the ideal grasping posture, even for the most state-of-the-art grippers. In this article, we presented a reconfigurable pneumatic gripper with fingers that could be set in various configurations, such as hooking, supporting, closuring, and pinching. Each…
▽ More
Grasping object,whether they are flat, round, or narrow and whether they have regular or irregular shapes,introduces difficulties in determining the ideal grasping posture, even for the most state-of-the-art grippers. In this article, we presented a reconfigurable pneumatic gripper with fingers that could be set in various configurations, such as hooking, supporting, closuring, and pinching. Each finger incorporates a dexterous joint, a rotating joint, and a customized plug-and-play visuotactile sensor, the DigiTac-v1.5, to control manipulation in real time. We propose a tactile kernel density manipulation strategy for simple and versatile control, including detecting grasp stability, responding to disturbances and guiding dexterous manipulations. We develop a double closed-loop control system that separately focuses on secure grasping and task management, demonstrated with tasks that highlight the capabilities above. The gripper is relatively easy to fabricate and customize, offering a promising and extensible way to combine soft dexterity and tactile sensing for diverse applications in robotic manipulation.
△ Less
Submitted 5 May, 2024;
originally announced May 2024.
-
Snap-it, Tap-it, Splat-it: Tactile-Informed 3D Gaussian Splatting for Reconstructing Challenging Surfaces
Authors:
Mauro Comi,
Alessio Tonioni,
Max Yang,
Jonathan Tremblay,
Valts Blukis,
Yijiong Lin,
Nathan F. Lepora,
Laurence Aitchison
Abstract:
Touch and vision go hand in hand, mutually enhancing our ability to understand the world. From a research perspective, the problem of mixing touch and vision is underexplored and presents interesting challenges. To this end, we propose Tactile-Informed 3DGS, a novel approach that incorporates touch data (local depth maps) with multi-view vision data to achieve surface reconstruction and novel view…
▽ More
Touch and vision go hand in hand, mutually enhancing our ability to understand the world. From a research perspective, the problem of mixing touch and vision is underexplored and presents interesting challenges. To this end, we propose Tactile-Informed 3DGS, a novel approach that incorporates touch data (local depth maps) with multi-view vision data to achieve surface reconstruction and novel view synthesis. Our method optimises 3D Gaussian primitives to accurately model the object's geometry at points of contact. By creating a framework that decreases the transmittance at touch locations, we achieve a refined surface reconstruction, ensuring a uniformly smooth depth map. Touch is particularly useful when considering non-Lambertian objects (e.g. shiny or reflective surfaces) since contemporary methods tend to fail to reconstruct with fidelity specular highlights. By combining vision and tactile sensing, we achieve more accurate geometry reconstructions with fewer images than prior methods. We conduct evaluation on objects with glossy and reflective surfaces and demonstrate the effectiveness of our approach, offering significant improvements in reconstruction quality.
△ Less
Submitted 29 March, 2024;
originally announced March 2024.
-
Pose and shear-based tactile servoing
Authors:
John Lloyd,
Nathan F. Lepora
Abstract:
Tactile servoing is an important technique because it enables robots to manipulate objects with precision and accuracy while adapting to changes in their environments in real-time. One approach for tactile servo control with high-resolution soft tactile sensors is to estimate the contact pose relative to an object surface using a convolutional neural network (CNN) for use as a feedback signal. In…
▽ More
Tactile servoing is an important technique because it enables robots to manipulate objects with precision and accuracy while adapting to changes in their environments in real-time. One approach for tactile servo control with high-resolution soft tactile sensors is to estimate the contact pose relative to an object surface using a convolutional neural network (CNN) for use as a feedback signal. In this paper, we investigate how the surface pose estimation model can be extended to include shear, and utilize these combined pose-and-shear models to develop a tactile robotic system that can be programmed for diverse non-prehensile manipulation tasks, such as object tracking, surface following, single-arm object pushing and dual-arm object pushing. In doing this, two technical challenges had to be overcome. Firstly, the use of tactile data that includes shear-induced slippage can lead to error-prone estimates unsuitable for accurate control, and so we modified the CNN into a Gaussian-density neural network and used a discriminative Bayesian filter to improve the predictions with a state dynamics model that utilizes the robot kinematics. Secondly, to achieve smooth robot motion in 3D space while interacting with objects, we used SE(3) velocity-based servo control, which required re-deriving the Bayesian filter update equations using Lie group theory, as many standard assumptions do not hold for state variables defined on non-Euclidean manifolds. In future, we believe that pose and shear-based tactile servoing will enable many object manipulation tasks and the fully-dexterous utilization of multi-fingered tactile robot hands. Video: https://www.youtube.com/watch?v=xVs4hd34ek0
△ Less
Submitted 13 December, 2023;
originally announced December 2023.
-
TouchSDF: A DeepSDF Approach for 3D Shape Reconstruction using Vision-Based Tactile Sensing
Authors:
Mauro Comi,
Yijiong Lin,
Alex Church,
Alessio Tonioni,
Laurence Aitchison,
Nathan F. Lepora
Abstract:
Humans rely on their visual and tactile senses to develop a comprehensive 3D understanding of their physical environment. Recently, there has been a growing interest in exploring and manipulating objects using data-driven approaches that utilise high-resolution vision-based tactile sensors. However, 3D shape reconstruction using tactile sensing has lagged behind visual shape reconstruction because…
▽ More
Humans rely on their visual and tactile senses to develop a comprehensive 3D understanding of their physical environment. Recently, there has been a growing interest in exploring and manipulating objects using data-driven approaches that utilise high-resolution vision-based tactile sensors. However, 3D shape reconstruction using tactile sensing has lagged behind visual shape reconstruction because of limitations in existing techniques, including the inability to generalise over unseen shapes, the absence of real-world testing, and limited expressive capacity imposed by discrete representations. To address these challenges, we propose TouchSDF, a Deep Learning approach for tactile 3D shape reconstruction that leverages the rich information provided by a vision-based tactile sensor and the expressivity of the implicit neural representation DeepSDF. Our technique consists of two components: (1) a Convolutional Neural Network that maps tactile images into local meshes representing the surface at the touch location, and (2) an implicit neural function that predicts a signed distance function to extract the desired 3D shape. This combination allows TouchSDF to reconstruct smooth and continuous 3D shapes from tactile inputs in simulation and real-world settings, opening up research avenues for robust 3D-aware representations and improved multimodal perception in robotics. Code and supplementary material are available at: https://touchsdf.github.io/
△ Less
Submitted 21 November, 2023;
originally announced November 2023.
-
Attention for Robot Touch: Tactile Saliency Prediction for Robust Sim-to-Real Tactile Control
Authors:
Yijiong Lin,
Mauro Comi,
Alex Church,
Dandan Zhang,
Nathan F. Lepora
Abstract:
High-resolution tactile sensing can provide accurate information about local contact in contact-rich robotic tasks. However, the deployment of such tasks in unstructured environments remains under-investigated. To improve the robustness of tactile robot control in unstructured environments, we propose and study a new concept: \textit{tactile saliency} for robot touch, inspired by the human touch a…
▽ More
High-resolution tactile sensing can provide accurate information about local contact in contact-rich robotic tasks. However, the deployment of such tasks in unstructured environments remains under-investigated. To improve the robustness of tactile robot control in unstructured environments, we propose and study a new concept: \textit{tactile saliency} for robot touch, inspired by the human touch attention mechanism from neuroscience and the visual saliency prediction problem from computer vision. In analogy to visual saliency, this concept involves identifying key information in tactile images captured by a tactile sensor. While visual saliency datasets are commonly annotated by humans, manually labelling tactile images is challenging due to their counterintuitive patterns. To address this challenge, we propose a novel approach comprised of three interrelated networks: 1) a Contact Depth Network (ConDepNet), which generates a contact depth map to localize deformation in a real tactile image that contains target and noise features; 2) a Tactile Saliency Network (TacSalNet), which predicts a tactile saliency map to describe the target areas for an input contact depth map; 3) and a Tactile Noise Generator (TacNGen), which generates noise features to train the TacSalNet. Experimental results in contact pose estimation and edge-following in the presence of distractors showcase the accurate prediction of target features from real tactile images. Overall, our tactile saliency prediction approach gives robust sim-to-real tactile control in environments with unknown distractors. Project page: https://sites.google.com/view/tactile-saliency/.
△ Less
Submitted 2 August, 2023; v1 submitted 26 July, 2023;
originally announced July 2023.
-
Sim-to-Real Model-Based and Model-Free Deep Reinforcement Learning for Tactile Pushing
Authors:
Max Yang,
Yijiong Lin,
Alex Church,
John Lloyd,
Dandan Zhang,
David A. W. Barton,
Nathan F. Lepora
Abstract:
Object pushing presents a key non-prehensile manipulation problem that is illustrative of more complex robotic manipulation tasks. While deep reinforcement learning (RL) methods have demonstrated impressive learning capabilities using visual input, a lack of tactile sensing limits their capability for fine and reliable control during manipulation. Here we propose a deep RL approach to object pushi…
▽ More
Object pushing presents a key non-prehensile manipulation problem that is illustrative of more complex robotic manipulation tasks. While deep reinforcement learning (RL) methods have demonstrated impressive learning capabilities using visual input, a lack of tactile sensing limits their capability for fine and reliable control during manipulation. Here we propose a deep RL approach to object pushing using tactile sensing without visual input, namely tactile pushing. We present a goal-conditioned formulation that allows both model-free and model-based RL to obtain accurate policies for pushing an object to a goal. To achieve real-world performance, we adopt a sim-to-real approach. Our results demonstrate that it is possible to train on a single object and a limited sample of goals to produce precise and reliable policies that can generalize to a variety of unseen objects and pushing scenarios without domain randomization. We experiment with the trained agents in harsh pushing conditions, and show that with significantly more training samples, a model-free policy can outperform a model-based planner, generating shorter and more reliable pushing trajectories despite large disturbances. The simplicity of our training environment and effective real-world performance highlights the value of rich tactile information for fine manipulation. Code and videos are available at https://sites.google.com/view/tactile-rl-pushing/.
△ Less
Submitted 26 July, 2023;
originally announced July 2023.
-
Bi-Touch: Bimanual Tactile Manipulation with Sim-to-Real Deep Reinforcement Learning
Authors:
Yijiong Lin,
Alex Church,
Max Yang,
Haoran Li,
John Lloyd,
Dandan Zhang,
Nathan F. Lepora
Abstract:
Bimanual manipulation with tactile feedback will be key to human-level robot dexterity. However, this topic is less explored than single-arm settings, partly due to the availability of suitable hardware along with the complexity of designing effective controllers for tasks with relatively large state-action spaces. Here we introduce a dual-arm tactile robotic system (Bi-Touch) based on the Tactile…
▽ More
Bimanual manipulation with tactile feedback will be key to human-level robot dexterity. However, this topic is less explored than single-arm settings, partly due to the availability of suitable hardware along with the complexity of designing effective controllers for tasks with relatively large state-action spaces. Here we introduce a dual-arm tactile robotic system (Bi-Touch) based on the Tactile Gym 2.0 setup that integrates two affordable industrial-level robot arms with low-cost high-resolution tactile sensors (TacTips). We present a suite of bimanual manipulation tasks tailored towards tactile feedback: bi-pushing, bi-reorienting and bi-gathering. To learn effective policies, we introduce appropriate reward functions for these tasks and propose a novel goal-update mechanism with deep reinforcement learning. We also apply these policies to real-world settings with a tactile sim-to-real approach. Our analysis highlights and addresses some challenges met during the sim-to-real application, e.g. the learned policy tended to squeeze an object in the bi-reorienting task due to the sim-to-real gap. Finally, we demonstrate the generalizability and robustness of this system by experimenting with different unseen objects with applied perturbations in the real world. Code and videos are available at https://sites.google.com/view/bi-touch/.
△ Less
Submitted 12 July, 2023;
originally announced July 2023.
-
Learning Fine Pinch-Grasp Skills using Tactile Sensing from A Few Real-world Demonstrations
Authors:
Xiaofeng Mao,
Yucheng Xu,
Ruoshi Wen,
Mohammadreza Kasaei,
Wanming Yu,
Efi Psomopoulou,
Nathan F. Lepora,
Zhibin Li
Abstract:
Imitation learning for robot dexterous manipulation, especially with a real robot setup, typically requires a large number of demonstrations. In this paper, we present a data-efficient learning from demonstration framework which exploits the use of rich tactile sensing data and achieves fine bimanual pinch grasping. Specifically, we employ a convolutional autoencoder network that can effectively e…
▽ More
Imitation learning for robot dexterous manipulation, especially with a real robot setup, typically requires a large number of demonstrations. In this paper, we present a data-efficient learning from demonstration framework which exploits the use of rich tactile sensing data and achieves fine bimanual pinch grasping. Specifically, we employ a convolutional autoencoder network that can effectively extract and encode high-dimensional tactile information. Further, We develop a framework that achieves efficient multi-sensor fusion for imitation learning, allowing the robot to learn contact-aware sensorimotor skills from demonstrations. Our comparision study against the framework without using encoded tactile features highlighted the effectiveness of incorporating rich contact information, which enabled dexterous bimanual grasping with active contact searching. Extensive experiments demonstrated the robustness of the fine pinch grasp policy directly learned from few-shot demonstration, including grasping of the same object with different initial poses, generalizing to ten unseen new objects, robust and firm grasping against external pushes, as well as contact-aware and reactive re-grasping in case of dropping objects under very large perturbations. Furthermore, the saliency map analysis method is used to describe weight distribution across various modalities during pinch grasping, confirming the effectiveness of our framework at leveraging multimodal information.
△ Less
Submitted 17 March, 2024; v1 submitted 10 July, 2023;
originally announced July 2023.
-
TacMMs: Tactile Mobile Manipulators for Warehouse Automation
Authors:
Zhuochao He,
Xuyang Zhang,
Simon Jones,
Sabine Hauert,
Dandan Zhang,
Nathan F. Lepora
Abstract:
Multi-robot platforms are playing an increasingly important role in warehouse automation for efficient goods transport. This paper proposes a novel customization of a multi-robot system, called Tactile Mobile Manipulators (TacMMs). Each TacMM integrates a soft optical tactile sensor and a mobile robot with a load-lifting mechanism, enabling cooperative transportation in tasks requiring coordinated…
▽ More
Multi-robot platforms are playing an increasingly important role in warehouse automation for efficient goods transport. This paper proposes a novel customization of a multi-robot system, called Tactile Mobile Manipulators (TacMMs). Each TacMM integrates a soft optical tactile sensor and a mobile robot with a load-lifting mechanism, enabling cooperative transportation in tasks requiring coordinated physical interaction. More specifically, we mount the TacTip (biomimetic optical tactile sensor) on the Distributed Organisation and Transport System (DOTS) mobile robot. The tactile information then helps the mobile robots adjust the relative robot-object pose, thereby increasing the efficiency of load-lifting tasks. This study compares the performance of using two TacMMs with tactile perception with traditional vision-based pose adjustment for load-lifting. The results show that the average success rate of the TacMMs (66%) is improved over a purely visual-based method (34%), with a larger improvement when the mass of the load was non-uniformly distributed. Although this initial study considers two TacMMs, we expect the benefits of tactile perception to extend to multiple mobile robots. Website: https://sites.google.com/view/tacmms
△ Less
Submitted 29 June, 2023;
originally announced June 2023.
-
Tactile-Driven Gentle Grasping for Human-Robot Collaborative Tasks
Authors:
Christopher J. Ford,
Haoran Li,
John Lloyd,
Manuel G. Catalano,
Matteo Bianchi,
Efi Psomopoulou,
Nathan F. Lepora
Abstract:
This paper presents a control scheme for force sensitive, gentle grasping with a Pisa/IIT anthropomorphic SoftHand equipped with a miniaturised version of the TacTip optical tactile sensor on all five fingertips. The tactile sensors provide high-resolution information about a grasp and how the fingers interact with held objects. We first describe a series of hardware developments for performing as…
▽ More
This paper presents a control scheme for force sensitive, gentle grasping with a Pisa/IIT anthropomorphic SoftHand equipped with a miniaturised version of the TacTip optical tactile sensor on all five fingertips. The tactile sensors provide high-resolution information about a grasp and how the fingers interact with held objects. We first describe a series of hardware developments for performing asynchronous sensor data acquisition and processing, resulting in a fast control loop sufficient for real-time grasp control. We then develop a novel grasp controller that uses tactile feedback from all five fingertip sensors simultaneously to gently and stably grasp 43 objects of varying geometry and stiffness, which is then applied to a human-to-robot handover task. These developments open the door to more advanced manipulation with underactuated hands via fast reflexive control using high-resolution tactile sensing.
△ Less
Submitted 16 March, 2023;
originally announced March 2023.
-
Tac-VGNN: A Voronoi Graph Neural Network for Pose-Based Tactile Servoing
Authors:
Wen Fan,
Max Yang,
Yifan Xing,
Nathan F. Lepora,
Dandan Zhang
Abstract:
Tactile pose estimation and tactile servoing are fundamental capabilities of robot touch. Reliable and precise pose estimation can be provided by applying deep learning models to high-resolution optical tactile sensors. Given the recent successes of Graph Neural Network (GNN) and the effectiveness of Voronoi features, we developed a Tactile Voronoi Graph Neural Network (Tac-VGNN) to achieve reliab…
▽ More
Tactile pose estimation and tactile servoing are fundamental capabilities of robot touch. Reliable and precise pose estimation can be provided by applying deep learning models to high-resolution optical tactile sensors. Given the recent successes of Graph Neural Network (GNN) and the effectiveness of Voronoi features, we developed a Tactile Voronoi Graph Neural Network (Tac-VGNN) to achieve reliable pose-based tactile servoing relying on a biomimetic optical tactile sensor (TacTip). The GNN is well suited to modeling the distribution relationship between shear motions of the tactile markers, while the Voronoi diagram supplements this with area-based tactile features related to contact depth. The experiment results showed that the Tac-VGNN model can help enhance data interpretability during graph generation and model training efficiency significantly than CNN-based methods. It also improved pose estimation accuracy along vertical depth by 28.57% over vanilla GNN without Voronoi features and achieved better performance on the real surface following tasks with smoother robot control trajectories. For more project details, please view our website: https://sites.google.com/view/tac-vgnn/home
△ Less
Submitted 5 March, 2023;
originally announced March 2023.
-
Semi-Supervised Disentanglement of Tactile Contact~Geometry from Sliding-Induced Shear
Authors:
Anupam K. Gupta,
Alex Church,
Nathan F. Lepora
Abstract:
The sense of touch is fundamental to human dexterity. When mimicked in robotic touch, particularly by use of soft optical tactile sensors, it suffers from distortion due to motion-dependent shear. This complicates tactile tasks like shape reconstruction and exploration that require information about contact geometry. In this work, we pursue a semi-supervised approach to remove shear while preservi…
▽ More
The sense of touch is fundamental to human dexterity. When mimicked in robotic touch, particularly by use of soft optical tactile sensors, it suffers from distortion due to motion-dependent shear. This complicates tactile tasks like shape reconstruction and exploration that require information about contact geometry. In this work, we pursue a semi-supervised approach to remove shear while preserving contact-only information. We validate our approach by showing a match between the model-generated unsheared images with their counterparts from vertically tapping onto the object. The model-generated unsheared images give faithful reconstruction of contact-geometry otherwise masked by shear, along with robust estimation of object pose then used for sliding exploration and full reconstruction of several planar shapes. We show that our semi-supervised approach achieves comparable performance to its fully supervised counterpart across all validation tasks with an order of magnitude less supervision. The semi-supervised method is thus more computational and labeled sample-efficient. We expect it will have broad applicability to wide range of complex tactile exploration and manipulation tasks performed via a shear-sensitive sense of touch.
△ Less
Submitted 26 August, 2022;
originally announced August 2022.
-
Tactile Gym 2.0: Sim-to-real Deep Reinforcement Learning for Comparing Low-cost High-Resolution Robot Touch
Authors:
Yijiong Lin,
John Lloyd,
Alex Church,
Nathan F. Lepora
Abstract:
High-resolution optical tactile sensors are increasingly used in robotic learning environments due to their ability to capture large amounts of data directly relating to agent-environment interaction. However, there is a high barrier of entry to research in this area due to the high cost of tactile robot platforms, specialised simulation software, and sim-to-real methods that lack generality acros…
▽ More
High-resolution optical tactile sensors are increasingly used in robotic learning environments due to their ability to capture large amounts of data directly relating to agent-environment interaction. However, there is a high barrier of entry to research in this area due to the high cost of tactile robot platforms, specialised simulation software, and sim-to-real methods that lack generality across different sensors. In this letter we extend the Tactile Gym simulator to include three new optical tactile sensors (TacTip, DIGIT and DigiTac) of the two most popular types, Gelsight-style (image-shading based) and TacTip-style (marker based). We demonstrate that a single sim-to-real approach can be used with these three different sensors to achieve strong real-world performance despite the significant differences between real tactile images. Additionally, we lower the barrier of entry to the proposed tasks by adapting them to an inexpensive 4-DoF robot arm, further enabling the dissemination of this benchmark. We validate the extended environment on three physically-interactive tasks requiring a sense of touch: object pushing, edge following and surface following. The results of our experimental validation highlight some differences between these sensors, which may help future researchers select and customize the physical characteristics of tactile sensors for different manipulations scenarios.
△ Less
Submitted 27 July, 2022; v1 submitted 21 July, 2022;
originally announced July 2022.
-
Mapping Mid-air Haptics with a Low-cost Tactile Robot
Authors:
Noor Alakhawand,
William Frier,
Nathan F. Lepora
Abstract:
Mid-air haptics create a new mode of feedback to allow people to feel tactile sensations in the air. Ultrasonic arrays focus acoustic radiation pressure in space, to induce tactile sensation from the resulting skin deflection. In this work, we present a low-cost tactile robot to test mid-air haptics. By combining a desktop robot arm with a 3D-printed biomimetic tactile sensor, we developed a syste…
▽ More
Mid-air haptics create a new mode of feedback to allow people to feel tactile sensations in the air. Ultrasonic arrays focus acoustic radiation pressure in space, to induce tactile sensation from the resulting skin deflection. In this work, we present a low-cost tactile robot to test mid-air haptics. By combining a desktop robot arm with a 3D-printed biomimetic tactile sensor, we developed a system that can sense, map, and visualize mid-air haptic sensations created by an ultrasonic transducer array. We evaluate our tactile robot by testing it on a variety of mid-air haptic stimuli, including unmodulated and modulated focal points that create a range of haptic shapes. We compare the mapping of the stimuli to another method used to test mid-air haptics: Laser Doppler Vibrometry, highlighting the advantages of the tactile robot including far lower cost, a small lightweight form-factor, and ease-of-use. Overall, these findings indicate our method has multiple benefits for sensing mid-air haptics and opens up new possibilities for expanding the testing to better emulate human haptic perception.
△ Less
Submitted 4 July, 2022;
originally announced July 2022.
-
DigiTac: A DIGIT-TacTip Hybrid Tactile Sensor for Comparing Low-Cost High-Resolution Robot Touch
Authors:
Nathan F. Lepora,
Yijiong Lin,
Ben Money-Coomes,
John Lloyd
Abstract:
Deep learning combined with high-resolution tactile sensing could lead to highly capable dexterous robots. However, progress is slow because of the specialist equipment and expertise. The DIGIT tactile sensor offers low-cost entry to high-resolution touch using GelSight-type sensors. Here we customize the DIGIT to have a 3D-printed sensing surface based on the TacTip family of soft biomimetic opti…
▽ More
Deep learning combined with high-resolution tactile sensing could lead to highly capable dexterous robots. However, progress is slow because of the specialist equipment and expertise. The DIGIT tactile sensor offers low-cost entry to high-resolution touch using GelSight-type sensors. Here we customize the DIGIT to have a 3D-printed sensing surface based on the TacTip family of soft biomimetic optical tactile sensors. The DIGIT-TacTip (DigiTac) enables direct comparison between these distinct tactile sensor types. For this comparison, we introduce a tactile robot system comprising a desktop arm, mounts and 3D-printed test objects. We use tactile servo control with a PoseNet deep learning model to compare the DIGIT, DigiTac and TacTip for edge- and surface-following over 3D-shapes. All three sensors performed similarly at pose prediction, but their constructions led to differing performances at servo control, offering guidance for researchers selecting or innovating tactile sensors. All hardware and software for reproducing this study will be openly released. Project website: www.lepora.com/digitac. Project repository: www.github.com/nlepora/digitac-design.
△ Less
Submitted 18 July, 2022; v1 submitted 27 June, 2022;
originally announced June 2022.
-
BRL/Pisa/IIT SoftHand: A Low-cost, 3D-Printed, Underactuated, Tendon-Driven Hand with Soft and Adaptive Synergies
Authors:
Haoran Li,
Christopher J. Ford,
Matteo Bianchi,
Manuel G. Catalano,
Efi Psomopoulou,
Nathan F. Lepora
Abstract:
This paper introduces the BRL/Pisa/IIT (BPI) SoftHand: a single actuator-driven, low-cost, 3D-printed, tendon-driven, underactuated robot hand that can be used to perform a range of grasping tasks. Based on the adaptive synergies of the Pisa/IIT SoftHand, we design a new joint system and tendon routing to facilitate the inclusion of both soft and adaptive synergies, which helps us balance durabili…
▽ More
This paper introduces the BRL/Pisa/IIT (BPI) SoftHand: a single actuator-driven, low-cost, 3D-printed, tendon-driven, underactuated robot hand that can be used to perform a range of grasping tasks. Based on the adaptive synergies of the Pisa/IIT SoftHand, we design a new joint system and tendon routing to facilitate the inclusion of both soft and adaptive synergies, which helps us balance durability, affordability and grasping performance of the hand. The focus of this work is on the design, simulation, synergies and grasping tests of this SoftHand. The novel phalanges are designed and printed based on linkages, gear pairs and geometric restraint mechanisms, and can be applied to most tendon-driven robotic hands. We show that the robot hand can successfully grasp and lift various target objects and adapt to hold complex geometric shapes, reflecting the successful adoption of the soft and adaptive synergies. We intend to open-source the design of the hand so that it can be built cheaply on a home 3D-printer. For more detail: https://sites.google.com/view/bpi-softhandtactile-group-bri/brlpisaiit-softhand-design
△ Less
Submitted 25 June, 2022;
originally announced June 2022.
-
Artificial SA-I, RA-I and RA-II/Vibrotactile Afferents for Tactile Sensing of Texture
Authors:
Nicholas Pestell,
Nathan F. Lepora
Abstract:
Robot touch can benefit from how humans perceive tactile textural information, from the stimulation mode to which tactile channels respond, then the tactile cues and encoding. Using a soft biomimetic tactile sensor (the TacTip) based on the physiology of the dermal-epidermal boundary, we construct two biomimetic tactile channels based on slowly-adapting SA-I and rapidly-adapting RA-I afferents, an…
▽ More
Robot touch can benefit from how humans perceive tactile textural information, from the stimulation mode to which tactile channels respond, then the tactile cues and encoding. Using a soft biomimetic tactile sensor (the TacTip) based on the physiology of the dermal-epidermal boundary, we construct two biomimetic tactile channels based on slowly-adapting SA-I and rapidly-adapting RA-I afferents, and introduce an additional sub-modality for vibrotactile information with an embedded microphone interpreted as an artificial RA-II channel. These artificial tactile channels are stimulated dynamically with a set of 13 artificial rigid textures comprising raised-bump patterns on a rotating drum that vary systematically in roughness. Methods employing spatial, spatio-temporal and temporal codes are assessed for texture classification insensitive to stimulation speed. We find: (i) spatially-encoded frictional cues provide a salient representation of texture; (ii) a simple transformation of spatial tactile features to model natural afferent responses improves the temporal coding; and (iii) the harmonic structure of induced vibrations provides a pertinent code for speed-invariant texture classification. Just as human touch relies on an interplay between slowly-adapting (SA-I), rapidly-adapting (RA-I) and vibrotactile (RA-II) channels, this tripartite structure may be needed for future robot applications with human-like dexterity, from prosthetics to materials testing, handling and manipulation.
△ Less
Submitted 5 April, 2022;
originally announced April 2022.
-
Tactile Image-to-Image Disentanglement of Contact Geometry from Motion-Induced Shear
Authors:
Anupam K. Gupta,
Laurence Aitchison,
Nathan F. Lepora
Abstract:
Robotic touch, particularly when using soft optical tactile sensors, suffers from distortion caused by motion-dependent shear. The manner in which the sensor contacts a stimulus is entangled with the tactile information about the geometry of the stimulus. In this work, we propose a supervised convolutional deep neural network model that learns to disentangle, in the latent space, the components of…
▽ More
Robotic touch, particularly when using soft optical tactile sensors, suffers from distortion caused by motion-dependent shear. The manner in which the sensor contacts a stimulus is entangled with the tactile information about the geometry of the stimulus. In this work, we propose a supervised convolutional deep neural network model that learns to disentangle, in the latent space, the components of sensor deformations caused by contact geometry from those due to sliding-induced shear. The approach is validated by reconstructing unsheared tactile images from sheared images and showing they match unsheared tactile images collected with no sliding motion. In addition, the unsheared tactile images give a faithful reconstruction of the contact geometry that is not possible from the sheared data, and robust estimation of the contact pose that can be used for servo control sliding around various 2D shapes. Finally, the contact geometry reconstruction in conjunction with servo control sliding were used for faithful full object reconstruction of various 2D shapes. The methods have broad applicability to deep learning models for robots with a shear-sensitive sense of touch.
△ Less
Submitted 8 September, 2021;
originally announced September 2021.
-
Artificial SA-I and RA-I Afferents for Tactile Sensing of Ridges and Gratings
Authors:
Nicholas Pestell,
Thom Griffith,
Nathan F. Lepora
Abstract:
For robot touch to converge with the human sense of touch, artificial transduction should involve biologically-plausible population codes analogous to those of natural afferents. Using a biomimetic tactile sensor with 3d-printed skin based on the dermal-epidermal boundary, we propose two novel feature sets to mimic slowly-adapting and rapidly-adapting type-I tactile mechanoreceptor function. Their…
▽ More
For robot touch to converge with the human sense of touch, artificial transduction should involve biologically-plausible population codes analogous to those of natural afferents. Using a biomimetic tactile sensor with 3d-printed skin based on the dermal-epidermal boundary, we propose two novel feature sets to mimic slowly-adapting and rapidly-adapting type-I tactile mechanoreceptor function. Their plausibility is tested with three classic experiments from the study of natural touch: impingement on a flat plate to probe adaptation and spatial modulation; stimulation by spatially-complex ridged stimuli to probe single afferent responses; and perception of grating orientation to probe the population response. Our results show a match between artificial and natural afferent responses in their sensitivity to edges and gaps; likewise, the human and robot psychometric functions match for grating orientation. These findings could benefit robot manipulation, prosthetics and the neurophysiology of touch.
△ Less
Submitted 5 April, 2022; v1 submitted 5 July, 2021;
originally announced July 2021.
-
Tactile Sim-to-Real Policy Transfer via Real-to-Sim Image Translation
Authors:
Alex Church,
John Lloyd,
Raia Hadsell,
Nathan F. Lepora
Abstract:
Simulation has recently become key for deep reinforcement learning to safely and efficiently acquire general and complex control policies from visual and proprioceptive inputs. Tactile information is not usually considered despite its direct relation to environment interaction. In this work, we present a suite of simulated environments tailored towards tactile robotics and reinforcement learning.…
▽ More
Simulation has recently become key for deep reinforcement learning to safely and efficiently acquire general and complex control policies from visual and proprioceptive inputs. Tactile information is not usually considered despite its direct relation to environment interaction. In this work, we present a suite of simulated environments tailored towards tactile robotics and reinforcement learning. A simple and fast method of simulating optical tactile sensors is provided, where high-resolution contact geometry is represented as depth images. Proximal Policy Optimisation (PPO) is used to learn successful policies across all considered tasks. A data-driven approach enables translation of the current state of a real tactile sensor to corresponding simulated depth images. This policy is implemented within a real-time control loop on a physical robot to demonstrate zero-shot sim-to-real policy transfer on several physically-interactive tasks requiring a sense of touch.
△ Less
Submitted 31 October, 2021; v1 submitted 16 June, 2021;
originally announced June 2021.
-
Probabilistic Discriminative Models Address the Tactile Perceptual Aliasing Problem
Authors:
John Lloyd,
Yijiong Lin,
Nathan F. Lepora
Abstract:
In this paper, our aim is to highlight Tactile Perceptual Aliasing as a problem when using deep neural networks and other discriminative models. Perceptual aliasing will arise wherever a physical variable extracted from tactile data is subject to ambiguity between stimuli that are physically distinct. Here we address this problem using a probabilistic discriminative model implemented as a 5-compon…
▽ More
In this paper, our aim is to highlight Tactile Perceptual Aliasing as a problem when using deep neural networks and other discriminative models. Perceptual aliasing will arise wherever a physical variable extracted from tactile data is subject to ambiguity between stimuli that are physically distinct. Here we address this problem using a probabilistic discriminative model implemented as a 5-component mixture density network comprised of a deep neural network that predicts the parameters of a Gaussian mixture model. We show that discriminative regression models such as deep neural networks and Gaussian process regression perform poorly on aliased data, only making accurate predictions when the sources of aliasing are removed. In contrast, the mixture density network identifies aliased data with improved prediction accuracy. The uncertain predictions of the model form patterns that are consistent with the various sources of perceptual ambiguity. In our view, perceptual aliasing will become an unavoidable issue for robot touch as the field progresses to training robots that act in uncertain and unstructured environments, such as with deep reinforcement learning.
△ Less
Submitted 3 June, 2021;
originally announced June 2021.
-
A Robust Controller for Stable 3D Pinching using Tactile Sensing
Authors:
Efi Psomopoulou,
Nicholas Pestell,
Fotios Papadopoulos,
John Lloyd,
Zoe Doulgeri,
Nathan F. Lepora
Abstract:
This paper proposes a controller for stable grasping of unknown-shaped objects by two robotic fingers with tactile fingertips. The grasp is stabilised by rolling the fingertips on the contact surface and applying a desired grasping force to reach an equilibrium state. The validation is both in simulation and on a fully-actuated robot hand (the Shadow Modular Grasper) fitted with custom-built optic…
▽ More
This paper proposes a controller for stable grasping of unknown-shaped objects by two robotic fingers with tactile fingertips. The grasp is stabilised by rolling the fingertips on the contact surface and applying a desired grasping force to reach an equilibrium state. The validation is both in simulation and on a fully-actuated robot hand (the Shadow Modular Grasper) fitted with custom-built optical tactile sensors (based on the BRL TacTip). The controller requires the orientations of the contact surfaces, which are estimated by regressing a deep convolutional neural network over the tactile images. Overall, the grasp system is demonstrated to achieve stable equilibrium poses on various objects ranging in shape and softness, with the system being robust to perturbations and measurement errors. This approach also has promise to extend beyond grasping to stable in-hand object manipulation with multiple fingers.
△ Less
Submitted 1 July, 2021; v1 submitted 2 June, 2021;
originally announced June 2021.
-
Soft Biomimetic Optical Tactile Sensing with the TacTip: A Review
Authors:
Nathan F. Lepora
Abstract:
Reproducing the capabilities of the human sense of touch in machines is an important step in enabling robot manipulation to have the ease of human dexterity. A combination of robotic technologies will be needed, including soft robotics, biomimetics and the high-resolution sensing offered by optical tactile sensors. This combination is considered here as a SoftBOT (Soft Biomimetic Optical Tactile)…
▽ More
Reproducing the capabilities of the human sense of touch in machines is an important step in enabling robot manipulation to have the ease of human dexterity. A combination of robotic technologies will be needed, including soft robotics, biomimetics and the high-resolution sensing offered by optical tactile sensors. This combination is considered here as a SoftBOT (Soft Biomimetic Optical Tactile) sensor. This article reviews the BRL TacTip as a prototypical example of such a sensor. Topics include the relation between artificial skin morphology and the transduction principles of human touch, the nature and benefits of tactile shear sensing, 3D printing for fabrication and integration into robot hands, the application of AI to tactile perception and control, and the recent step-change in capabilities due to deep learning. This review consolidates those advances from the past decade to indicate a path for robots to reach human-like dexterity.
△ Less
Submitted 22 July, 2021; v1 submitted 30 May, 2021;
originally announced May 2021.
-
Uncertainty-aware deep learning for robot touch: Application to Bayesian tactile servo control
Authors:
Manuel Floriano Vazquez,
Nathan F. Lepora
Abstract:
This work investigates uncertainty-aware deep learning (DL) in tactile robotics based on a general framework introduced recently for robot vision. For a test scenario, we consider optical tactile sensing in combination with DL to estimate the edge pose as a feedback signal to servo around various 2D test objects. We demonstrate that uncertainty-aware DL can improve the pose estimation over determi…
▽ More
This work investigates uncertainty-aware deep learning (DL) in tactile robotics based on a general framework introduced recently for robot vision. For a test scenario, we consider optical tactile sensing in combination with DL to estimate the edge pose as a feedback signal to servo around various 2D test objects. We demonstrate that uncertainty-aware DL can improve the pose estimation over deterministic DL methods. The system estimates the uncertainty associated with each prediction, which is used along with temporal coherency to improve the predictions via a Kalman filter, and hence improve the tactile servo control. The robot is able to robustly follow all of the presented contour shapes to reduce not only the error by a factor of two but also smooth the trajectory from the undesired noisy behaviour caused by previous deterministic networks. In our view, as the field of tactile robotics matures in its use of DL, the estimation of uncertainty will become a key component in the control of physically interactive tasks in complex environments.
△ Less
Submitted 29 April, 2021;
originally announced April 2021.
-
Towards integrated tactile sensorimotor control in anthropomorphic soft robotic hands
Authors:
Nathan F. Lepora,
Andrew Stinchcombe,
Chris Ford,
Alfred Brown,
John Lloyd,
Manuel G. Catalano,
Matteo Bianchi,
Benjamin Ward-Cherrier
Abstract:
In this work, we report on the integrated sensorimotor control of the Pisa/IIT SoftHand, an anthropomorphic soft robot hand designed around the principle of adaptive synergies, with the BRL tactile fingertip (TacTip), a soft biomimetic optical tactile sensor based on the human sense of touch. Our focus is how a sense of touch can be used to control an anthropomorphic hand with one degree of actuat…
▽ More
In this work, we report on the integrated sensorimotor control of the Pisa/IIT SoftHand, an anthropomorphic soft robot hand designed around the principle of adaptive synergies, with the BRL tactile fingertip (TacTip), a soft biomimetic optical tactile sensor based on the human sense of touch. Our focus is how a sense of touch can be used to control an anthropomorphic hand with one degree of actuation, based on an integration that respects the hand's mechanical functionality. We consider: (i) closed-loop tactile control to establish a light contact on an unknown held object, based on the structural similarity with an undeformed tactile image; and (ii) controlling the estimated pose of an edge feature of a held object, using a convolutional neural network approach developed for controlling other sensors in the TacTip family. Overall, this gives a foundation to endow soft robotic hands with human-like touch, with implications for autonomous grasping, manipulation, human-robot interaction and prosthetics. Supplemental video: https://youtu.be/ndsxj659bkQ
△ Less
Submitted 5 February, 2021;
originally announced February 2021.
-
Pose-Based Tactile Servoing: Controlled Soft Touch using Deep Learning
Authors:
Nathan F. Lepora,
John Lloyd
Abstract:
This article describes a new way of controlling robots using soft tactile sensors: pose-based tactile servo (PBTS) control. The basic idea is to embed a tactile perception model for estimating the sensor pose within a servo control loop that is applied to local object features such as edges and surfaces. PBTS control is implemented with a soft curved optical tactile sensor (the BRL TacTip) using a…
▽ More
This article describes a new way of controlling robots using soft tactile sensors: pose-based tactile servo (PBTS) control. The basic idea is to embed a tactile perception model for estimating the sensor pose within a servo control loop that is applied to local object features such as edges and surfaces. PBTS control is implemented with a soft curved optical tactile sensor (the BRL TacTip) using a convolutional neural network trained to be insensitive to shear. In consequence, robust and accurate controlled motion over various complex 3D objects is attained. First, we review tactile servoing and its relation to visual servoing, before formalising PBTS control. Then, we assess tactile servoing over a range of regular and irregular objects. Finally, we reflect on the relation to visual servo control and discuss how controlled soft touch gives a route towards human-like dexterity in robots.
△ Less
Submitted 2 July, 2021; v1 submitted 4 December, 2020;
originally announced December 2020.
-
Goal-Driven Robotic Pushing Using Tactile and Proprioceptive Feedback
Authors:
John Lloyd,
Nathan F. Lepora
Abstract:
In robots, nonprehensile manipulation operations such as pushing are a useful way of moving large, heavy or unwieldy objects, moving multiple objects at once, or reducing uncertainty in the location or pose of objects. In this study, we propose a reactive and adaptive method for robotic pushing that uses rich feedback from a high-resolution optical tactile sensor to control push movements instead…
▽ More
In robots, nonprehensile manipulation operations such as pushing are a useful way of moving large, heavy or unwieldy objects, moving multiple objects at once, or reducing uncertainty in the location or pose of objects. In this study, we propose a reactive and adaptive method for robotic pushing that uses rich feedback from a high-resolution optical tactile sensor to control push movements instead of relying on analytical or data-driven models of push interactions. Specifically, we use goal-driven tactile exploration to actively search for stable pushing configurations that cause the object to maintain its pose relative to the pusher while incrementally moving the pusher and object towards the target. We evaluate our method by pushing objects across planar and curved surfaces. For planar surfaces, we show that the method is accurate and robust to variations in initial contact position/angle, object shape and start position; for curved surfaces, the performance is degraded slightly. An immediate consequence of our work is that it shows that explicit models of push interactions might be sufficient but are not necessary for this type of task. It also raises the interesting question of which aspects of the system should be modelled to achieve the best performance and generalization across a wide range of scenarios. Finally, it highlights the importance of testing on non-planar surfaces and in other more complex environments when developing new methods for robotic pushing.
△ Less
Submitted 2 August, 2021; v1 submitted 3 December, 2020;
originally announced December 2020.
-
Spatio-temporal encoding improves neuromorphic tactile texture classification
Authors:
Anupam K. Gupta,
Andrei Nakagawa,
Nathan F. Lepora,
Nitish V. Thakor
Abstract:
With the increase in interest in deployment of robots in unstructured environments to work alongside humans, the development of human-like sense of touch for robots becomes important. In this work, we implement a multi-channel neuromorphic tactile system that encodes contact events as discrete spike events that mimic the behavior of slow adapting mechanoreceptors. We study the impact of informatio…
▽ More
With the increase in interest in deployment of robots in unstructured environments to work alongside humans, the development of human-like sense of touch for robots becomes important. In this work, we implement a multi-channel neuromorphic tactile system that encodes contact events as discrete spike events that mimic the behavior of slow adapting mechanoreceptors. We study the impact of information pooling across artificial mechanoreceptors on classification performance of spatially non-uniform naturalistic textures. We encoded the spatio-temporal activation patterns of mechanoreceptors through gray-level co-occurrence matrix computed from time-varying mean spiking rate-based tactile response volume. We found that this approach greatly improved texture classification in comparison to use of individual mechanoreceptor response alone. In addition, the performance was also more robust to changes in sliding velocity. The importance of exploiting precise spatial and temporal correlations between sensory channels is evident from the fact that on either removal of precise temporal information or altering of spatial structure of response pattern, a significant performance drop was observed. This study thus demonstrates the superiority of population coding approaches that can exploit the precise spatio-temporal information encoded in activation patterns of mechanoreceptor populations. It, therefore, makes an advance in the direction of development of bio-inspired tactile systems required for realistic touch applications in robotics and prostheses.
△ Less
Submitted 5 June, 2021; v1 submitted 27 October, 2020;
originally announced October 2020.
-
Slip detection for grasp stabilisation with a multi-fingered tactile robot hand
Authors:
Jasper W. James,
Nathan F. Lepora
Abstract:
Tactile sensing is used by humans when grasping to prevent us dropping objects. One key facet of tactile sensing is slip detection, which allows a gripper to know when a grasp is failing and take action to prevent an object being dropped. This study demonstrates the slip detection capabilities of the recently developed Tactile Model O (T-MO) by using support vector machines to detect slip and test…
▽ More
Tactile sensing is used by humans when grasping to prevent us dropping objects. One key facet of tactile sensing is slip detection, which allows a gripper to know when a grasp is failing and take action to prevent an object being dropped. This study demonstrates the slip detection capabilities of the recently developed Tactile Model O (T-MO) by using support vector machines to detect slip and test multiple slip scenarios including responding to the onset of slip in real time with eleven different objects in various grasps. We demonstrate the benefits of slip detection in grasping by testing two real-world scenarios: adding weight to destabilise a grasp and using slip detection to lift up objects at the first attempt. The T-MO is able to detect when an object is slipping, react to stabilise the grasp and be deployed in real-world scenarios. This shows the T-MO is a suitable platform for autonomous grasping by using reliable slip detection to ensure a stable grasp in unstructured environments. Supplementary video: https://youtu.be/wOwFHaiHuKY
△ Less
Submitted 5 October, 2020;
originally announced October 2020.
-
A Biomimetic Tactile Fingerprint Induces Incipient Slip
Authors:
Jasper W. James,
Stephen J. Redmond,
Nathan F. Lepora
Abstract:
We present a modified TacTip biomimetic optical tactile sensor design which demonstrates the ability to induce and detect incipient slip, as confirmed by recording the movement of markers on the sensor's external surface. Incipient slip is defined as slippage of part, but not all, of the contact surface between the sensor and object. The addition of ridges - which mimic the friction ridges in the…
▽ More
We present a modified TacTip biomimetic optical tactile sensor design which demonstrates the ability to induce and detect incipient slip, as confirmed by recording the movement of markers on the sensor's external surface. Incipient slip is defined as slippage of part, but not all, of the contact surface between the sensor and object. The addition of ridges - which mimic the friction ridges in the human fingertip - in a concentric ring pattern allowed for localised shear deformation to occur on the sensor surface for a significant duration prior to the onset of gross slip. By detecting incipient slip we were able to predict when several differently shaped objects were at risk of falling and prevent them from doing so. Detecting incipient slip is useful because a corrective action can be taken before slippage occurs across the entire contact area thus minimising the risk of objects been dropped.
△ Less
Submitted 16 August, 2020;
originally announced August 2020.
-
Walking on TacTip toes: A tactile sensing foot for walking robots
Authors:
Elizabeth A. Stone,
Nathan F. Lepora,
David A. W. Barton
Abstract:
Little research into tactile feet has been done for walking robots despite the benefits such feedback could give when walking on uneven terrain. This paper describes the development of a simple, robust and inexpensive tactile foot for legged robots based on a high-resolution biomimetic TacTip tactile sensor. Several design improvements were made to facilitate tactile sensing while walking, includi…
▽ More
Little research into tactile feet has been done for walking robots despite the benefits such feedback could give when walking on uneven terrain. This paper describes the development of a simple, robust and inexpensive tactile foot for legged robots based on a high-resolution biomimetic TacTip tactile sensor. Several design improvements were made to facilitate tactile sensing while walking, including the use of phosphorescent markers to remove the need for internal LED lighting. The usefulness of the foot is verified on a quadrupedal robot performing a beam walking task and it is found the sensor prevents the robot falling off the beam. Further, this capability also enables the robot to walk along the edge of a curved table. This tactile foot design can be easily modified for use with any legged robot, including much larger walking robots, enabling stable walking in challenging terrain.
△ Less
Submitted 12 August, 2020;
originally announced August 2020.
-
Deep Reinforcement Learning for Tactile Robotics: Learning to Type on a Braille Keyboard
Authors:
Alex Church,
John Lloyd,
Raia Hadsell,
Nathan F. Lepora
Abstract:
Artificial touch would seem well-suited for Reinforcement Learning (RL), since both paradigms rely on interaction with an environment. Here we propose a new environment and set of tasks to encourage development of tactile reinforcement learning: learning to type on a braille keyboard. Four tasks are proposed, progressing in difficulty from arrow to alphabet keys and from discrete to continuous act…
▽ More
Artificial touch would seem well-suited for Reinforcement Learning (RL), since both paradigms rely on interaction with an environment. Here we propose a new environment and set of tasks to encourage development of tactile reinforcement learning: learning to type on a braille keyboard. Four tasks are proposed, progressing in difficulty from arrow to alphabet keys and from discrete to continuous actions. A simulated counterpart is also constructed by sampling tactile data from the physical environment. Using state-of-the-art deep RL algorithms, we show that all of these tasks can be successfully learnt in simulation, and 3 out of 4 tasks can be learned on the real robot. A lack of sample efficiency currently makes the continuous alphabet task impractical on the robot. To the best of our knowledge, this work presents the first demonstration of successfully training deep RL agents in the real world using observations that exclusively consist of tactile images. To aid future research utilising this environment, the code for this project has been released along with designs of the braille keycaps for 3D printing and a guide for recreating the experiments. A brief video summary is also available at https://youtu.be/eNylCA2uE_E.
△ Less
Submitted 6 August, 2020;
originally announced August 2020.
-
Sim-to-Real Transfer for Optical Tactile Sensing
Authors:
Zihan Ding,
Nathan F. Lepora,
Edward Johns
Abstract:
Deep learning and reinforcement learning methods have been shown to enable learning of flexible and complex robot controllers. However, the reliance on large amounts of training data often requires data collection to be carried out in simulation, with a number of sim-to-real transfer methods being developed in recent years. In this paper, we study these techniques for tactile sensing using the Tac…
▽ More
Deep learning and reinforcement learning methods have been shown to enable learning of flexible and complex robot controllers. However, the reliance on large amounts of training data often requires data collection to be carried out in simulation, with a number of sim-to-real transfer methods being developed in recent years. In this paper, we study these techniques for tactile sensing using the TacTip optical tactile sensor, which consists of a deformable tip with a camera observing the positions of pins inside this tip. We designed a model for soft body simulation which was implemented using the Unity physics engine, and trained a neural network to predict the locations and angles of edges when in contact with the sensor. Using domain randomisation techniques for sim-to-real transfer, we show how this framework can be used to accurately predict edges with less than 1 mm prediction error in real-world testing, without any real-world data at all.
△ Less
Submitted 31 March, 2020;
originally announced April 2020.
-
Optimal Deep Learning for Robot Touch
Authors:
Nathan F. Lepora,
John Lloyd
Abstract:
This article illustrates the application of deep learning to robot touch by considering a basic yet fundamental capability: estimating the relative pose of part of an object in contact with a tactile sensor. We begin by surveying deep learning applied to tactile robotics, focussing on optical tactile sensors, which help bridge from deep learning for vision to touch. We then show how deep learning…
▽ More
This article illustrates the application of deep learning to robot touch by considering a basic yet fundamental capability: estimating the relative pose of part of an object in contact with a tactile sensor. We begin by surveying deep learning applied to tactile robotics, focussing on optical tactile sensors, which help bridge from deep learning for vision to touch. We then show how deep learning can be used to train accurate pose models of 3D surfaces and edges that are insensitive to nuisance variables such as motion-dependent shear. This involves including representative motions as unlabelled perturbations of the training data and using Bayesian optimization of the network and training hyperparameters to find the most accurate models. Accurate estimation of pose from touch will enable robots to safely and precisely control their physical interactions, underlying a wide range of object exploration and manipulation tasks.
△ Less
Submitted 28 March, 2020; v1 submitted 4 March, 2020;
originally announced March 2020.
-
NeuroTac: A Neuromorphic Optical Tactile Sensor applied to Texture Recognition
Authors:
Benjamin Ward-Cherrier,
Nicholas Pestell,
Nathan F. Lepora
Abstract:
Developing artificial tactile sensing capabilities that rival human touch is a long-term goal in robotics and prosthetics. Gradually more elaborate biomimetic tactile sensors are being developed and applied to grasping and manipulation tasks to help achieve this goal. Here we present the neuroTac, a novel neuromorphic optical tactile sensor. The neuroTac combines the biomimetic hardware design fro…
▽ More
Developing artificial tactile sensing capabilities that rival human touch is a long-term goal in robotics and prosthetics. Gradually more elaborate biomimetic tactile sensors are being developed and applied to grasping and manipulation tasks to help achieve this goal. Here we present the neuroTac, a novel neuromorphic optical tactile sensor. The neuroTac combines the biomimetic hardware design from the TacTip sensor which mimicks the layered papillae structure of human glabrous skin, with an event-based camera (DAVIS240, iniVation) and algorithms which transduce contact information in the form of spike trains. The performance of the sensor is evaluated on a texture classification task, with four spike coding methods being implemented and compared: Intensive, Spatial, Temporal and Spatiotemporal. We found timing-based coding methods performed with the highest accuracy over both artificial and natural textures. The spike-based output of the neuroTac could enable the development of biomimetic tactile perception algorithms in robotics as well as non-invasive and invasive haptic feedback methods in prosthetics.
△ Less
Submitted 1 March, 2020;
originally announced March 2020.
-
Learning to Live Life on the Edge: Online Learning for Data-Efficient Tactile Contour Following
Authors:
Elizabeth A. Stone,
Nathan F. Lepora,
David A. W. Barton
Abstract:
Tactile sensing has been used for a variety of robotic exploration and manipulation tasks but a common constraint is a requirement for a large amount of training data. This paper addresses the issue of data-efficiency by proposing a novel method for online learning based on a Gaussian Process Latent Variable Model (GP-LVM), whereby the robot learns from tactile data whilst performing a contour fol…
▽ More
Tactile sensing has been used for a variety of robotic exploration and manipulation tasks but a common constraint is a requirement for a large amount of training data. This paper addresses the issue of data-efficiency by proposing a novel method for online learning based on a Gaussian Process Latent Variable Model (GP-LVM), whereby the robot learns from tactile data whilst performing a contour following task thus enabling generalisation to a wide variety of tactile stimuli. The results show that contour following is successful with comparatively little data and is robust to novel stimuli. This work highlights that even with a simple learning architecture there are significant advantages to be gained in efficient and robust task performance by using latent variable models and online learning for tactile sensing tasks. This paves the way for a new generation of robust, fast, and data-efficient tactile systems.
△ Less
Submitted 12 August, 2020; v1 submitted 12 September, 2019;
originally announced September 2019.
-
Tactile Model O: Fabrication and testing of a 3d-printed, three-fingered tactile robot hand
Authors:
Jasper W. James,
Alex Church,
Luke Cramphorn,
Nathan F. Lepora
Abstract:
Bringing tactile sensation to robotic hands will allow for more effective grasping, along with the wide range of benefits of human-like touch. Here we present a 3D-printed, three-fingered tactile robot hand comprising an OpenHand Model O customized to house a TacTip soft biomimetic tactile sensor in the distal phalanx of each finger. We expect that combining the grasping capabilities of this under…
▽ More
Bringing tactile sensation to robotic hands will allow for more effective grasping, along with the wide range of benefits of human-like touch. Here we present a 3D-printed, three-fingered tactile robot hand comprising an OpenHand Model O customized to house a TacTip soft biomimetic tactile sensor in the distal phalanx of each finger. We expect that combining the grasping capabilities of this underactuated hand with sophisticated tactile sensing will result in an effective platform for robot hand research -- the Tactile Model O (T-MO). The design uses three JeVois machine vision systems, each comprising a miniature camera in the tactile fingertip with a processing module in the base of the hand. To evaluate the capabilities of the T-MO, we benchmark its grasping performance using the Gripper Assessment Benchmark on the YCB object set. Tactile sensing capabilities are evaluated by performing tactile object classification on 26 objects and predicting whether a grasp will successfully lift each object. Results are consistent with the state of the art, taking advantage of advances in deep learning applied to tactile image outputs. Overall, this work demonstrates that the T-MO is an effective platform for robot hand research and we expect it to open-up a range of applications in autonomous object handling. Supplemental video: https://youtu.be/RTcCpgffCrQ.
△ Less
Submitted 14 August, 2020; v1 submitted 17 July, 2019;
originally announced July 2019.
-
Shear-invariant Sliding Contact Perception with a Soft Tactile Sensor
Authors:
Kirsty Aquilina,
David A. W. Barton,
Nathan F. Lepora
Abstract:
Manipulation tasks often require robots to be continuously in contact with an object. Therefore tactile perception systems need to handle continuous contact data. Shear deformation causes the tactile sensor to output path-dependent readings in contrast to discrete contact readings. As such, in some continuous-contact tasks, sliding can be regarded as a disturbance over the sensor signal. Here we p…
▽ More
Manipulation tasks often require robots to be continuously in contact with an object. Therefore tactile perception systems need to handle continuous contact data. Shear deformation causes the tactile sensor to output path-dependent readings in contrast to discrete contact readings. As such, in some continuous-contact tasks, sliding can be regarded as a disturbance over the sensor signal. Here we present a shear-invariant perception method based on principal component analysis (PCA) which outputs the required information about the environment despite sliding motion. A compliant tactile sensor (the TacTip) is used to investigate continuous tactile contact. First, we evaluate the method offline using test data collected whilst the sensor slides over an edge. Then, the method is used within a contour-following task applied to 6 objects with varying curvatures; all contours are successfully traced. The method demonstrates generalisation capabilities and could underlie a more sophisticated controller for challenging manipulation or exploration tasks in unstructured environments. A video showing the work described in the paper can be found at https://youtu.be/wrTM61-pieU
△ Less
Submitted 2 May, 2019;
originally announced May 2019.
-
From pixels to percepts: Highly robust edge perception and contour following using deep learning and an optical biomimetic tactile sensor
Authors:
Nathan F. Lepora,
Alex Church,
Conrad De Kerckhove,
Raia Hadsell,
John Lloyd
Abstract:
Deep learning has the potential to have the impact on robot touch that it has had on robot vision. Optical tactile sensors act as a bridge between the subjects by allowing techniques from vision to be applied to touch. In this paper, we apply deep learning to an optical biomimetic tactile sensor, the TacTip, which images an array of papillae (pins) inside its sensing surface analogous to structure…
▽ More
Deep learning has the potential to have the impact on robot touch that it has had on robot vision. Optical tactile sensors act as a bridge between the subjects by allowing techniques from vision to be applied to touch. In this paper, we apply deep learning to an optical biomimetic tactile sensor, the TacTip, which images an array of papillae (pins) inside its sensing surface analogous to structures within human skin. Our main result is that the application of a deep CNN can give reliable edge perception and thus a robust policy for planning contact points to move around object contours. Robustness is demonstrated over several irregular and compliant objects with both tapping and continuous sliding, using a model trained only by tapping onto a disk. These results relied on using techniques to encourage generalization to tasks beyond which the model was trained. We expect this is a generic problem in practical applications of tactile sensing that deep learning will solve. A video demonstrating the approach can be found at https://www.youtube.com/watch?v=QHrGsG9AHts
△ Less
Submitted 6 February, 2019; v1 submitted 7 December, 2018;
originally announced December 2018.
-
TacWhiskers: Biomimetic optical tactile whiskered robots
Authors:
Nathan F. Lepora,
Martin Pearson,
Luke Cramphorn
Abstract:
Here we propose and investigate a novel vibrissal tactile sensor - the TacWhisker array - based on modifying a 3D-printed optical cutaneous (fingertip) tactile sensor - the TacTip. Two versions are considered: a static TacWhisker array analogous to immotile tactile vibrissae (e.g. rodent microvibrissae) and a dynamic TacWhisker array analogous to motile tactile vibrissae (e.g. rodent macrovibrissa…
▽ More
Here we propose and investigate a novel vibrissal tactile sensor - the TacWhisker array - based on modifying a 3D-printed optical cutaneous (fingertip) tactile sensor - the TacTip. Two versions are considered: a static TacWhisker array analogous to immotile tactile vibrissae (e.g. rodent microvibrissae) and a dynamic TacWhisker array analogous to motile tactile vibrissae (e.g. rodent macrovibrissae). Performance is assessed on an active object localization task. The whisking motion of the dynamic TacWhisker leads to millimetre-scale location perception, whereas perception with the static TacWhisker array is relatively poor when making dabbing contacts. The dynamic sensor output is dominated by a self-generated motion signal, which can be compensated by comparing to a reference signal. Overall, the TacWhisker arrays give a new class of tactile whiskered robots that benefit from being relatively inexpensive and customizable. Furthermore, the biomimetic basis for the TacWhiskers fits well with building an embodied model of the rodent sensory system for investigating animal perception. A video demonstrating this robot can be seen at https://www.youtube.com/watch?v=ksS177ep6yY
△ Less
Submitted 7 December, 2018; v1 submitted 30 September, 2018;
originally announced October 2018.
-
Principal Components of Touch
Authors:
Kirsty Aquilina,
David A. W. Barton,
Nathan F. Lepora
Abstract:
Our human sense of touch enables us to manipulate our surroundings; therefore, complex robotic manipulation will require artificial tactile sensing. Typically tactile sensor arrays are used in robotics, implying that a straightforward way of interpreting multidimensional data is required. In this paper we present a simple visualisation approach based on applying principal component analysis (PCA)…
▽ More
Our human sense of touch enables us to manipulate our surroundings; therefore, complex robotic manipulation will require artificial tactile sensing. Typically tactile sensor arrays are used in robotics, implying that a straightforward way of interpreting multidimensional data is required. In this paper we present a simple visualisation approach based on applying principal component analysis (PCA) to systematically collected sets of tactile data. We apply the visualisation approach to 4 different types of tactile sensor, encompassing fingertips and vibrissal arrays. The results show that PCA can reveal structure and regularities in the tactile data, which also permits the use of simple classifiers such as $k$-NN to achieve good inference. Additionally, the Euclidean distance in principal component space gives a measure of sensitivity, which can aid visualisation and also be used to find regions in the tactile input space where the sensor is able to perceive with higher accuracy. We expect that these observations will generalise, and thus offer the potential for novel control methods based on touch.
△ Less
Submitted 8 June, 2018; v1 submitted 21 May, 2018;
originally announced May 2018.
-
Voronoi Features for Tactile Sensing: Direct Inference of Pressure, Shear, and Contact Locations
Authors:
Luke Cramphorn,
John Lloyd,
Nathan F. Lepora
Abstract:
There are a wide range of features that tactile contact provides, each with different aspects of information that can be used for object grasping, manipulation, and perception. In this paper inference of some key tactile features, tip displacement, contact location, shear direction and magnitude, is demonstrated by introducing a novel method of transducing a third dimension to the sensor data via…
▽ More
There are a wide range of features that tactile contact provides, each with different aspects of information that can be used for object grasping, manipulation, and perception. In this paper inference of some key tactile features, tip displacement, contact location, shear direction and magnitude, is demonstrated by introducing a novel method of transducing a third dimension to the sensor data via Voronoi tessellation. The inferred features are displayed throughout the work in a new visualisation mode derived from the Voronoi tessellation; these visualisations create easier interpretation of data from an optical tactile sensor that measures local shear from displacement of internal pins (the TacTip). The output values of tip displacement and shear magnitude are calibrated to appropriate mechanical units and validate the direction of shear inferred from the sensor. We show that these methods can infer the direction of shear to $\sim$2.3$^{\circ}$ without the need for training a classifier or regressor. The approach demonstrated here will increase the versatility and generality of the sensors and thus allow sensor to be used in more unstructured and unknown environments, as well as improve the use of these tactile sensors in more complex systems such as robot hands.
△ Less
Submitted 20 May, 2018;
originally announced May 2018.
-
A comment on: `Some problems with calculating the quantum corrections to the classical 't Hooft-Polyakov monopole'
Authors:
Nathan F. Lepora
Abstract:
In a recent publication we noticed that the Hamiltonian density for fluctuations around the 't Hooft-Polyakov monopole appeared to be non-Hermitian. Here we show that when this Hamiltonian density is integrated into the Hamiltonian all non-Hermitian terms give a vanishing total contribution.
In a recent publication we noticed that the Hamiltonian density for fluctuations around the 't Hooft-Polyakov monopole appeared to be non-Hermitian. Here we show that when this Hamiltonian density is integrated into the Hamiltonian all non-Hermitian terms give a vanishing total contribution.
△ Less
Submitted 22 June, 2003;
originally announced June 2003.
-
Asymptotically embedded defects
Authors:
Nathan F. Lepora
Abstract:
For many cases, the conditions to fully embed a classical solution of one field theory within a larger theory cannot be met. Instead, we find it useful to embed only the solution's asymptotic fields as this relaxes the embedding constraints. Such asymptotically embedded defects have a simple classification that can be used to construct classical solutions in general field theories.
For many cases, the conditions to fully embed a classical solution of one field theory within a larger theory cannot be met. Instead, we find it useful to embed only the solution's asymptotic fields as this relaxes the embedding constraints. Such asymptotically embedded defects have a simple classification that can be used to construct classical solutions in general field theories.
△ Less
Submitted 2 October, 2002;
originally announced October 2002.
-
Some problems when calculating the quantum corrections to the classical 't Hooft-Polyakov monopole
Authors:
Nathan F. Lepora
Abstract:
We examine whether the analysis of quantum corrections for the kink soliton carries over to the 't Hooft-Polyakov monopole. For the kink, it is central that the quantum fluctuations are eigenmodes of a Hermitian operator. For the monopole, we show the analogous operator is not Hermitian. This property raises some questions about the quantization procedure for a 't Hooft-Polyakov monopole.
We examine whether the analysis of quantum corrections for the kink soliton carries over to the 't Hooft-Polyakov monopole. For the kink, it is central that the quantum fluctuations are eigenmodes of a Hermitian operator. For the monopole, we show the analogous operator is not Hermitian. This property raises some questions about the quantization procedure for a 't Hooft-Polyakov monopole.
△ Less
Submitted 23 July, 2002;
originally announced July 2002.
-
Some Simpler Analogues of the Dual Standard Model and their Relation to Bais' Generalisation of the Montenon-Olive Conjecture
Authors:
Nathan F. Lepora
Abstract:
We show that the correspondence between SU(5) monopoles and the elementary particles, which underlies the construction of a dual standard model, has some simpler analogues associated with the strong, weak and hypercharge interactions. We then discuss how these analogues relate to Bais' generalization of the Montenon-Olive conjecture and find the representations of the monopoles under the dual ga…
▽ More
We show that the correspondence between SU(5) monopoles and the elementary particles, which underlies the construction of a dual standard model, has some simpler analogues associated with the strong, weak and hypercharge interactions. We then discuss how these analogues relate to Bais' generalization of the Montenon-Olive conjecture and find the representations of the monopoles under the dual gauge group; these representations agree with those of the elementary particles.
△ Less
Submitted 23 July, 2002; v1 submitted 25 October, 2001;
originally announced October 2001.
-
Embedded Monopoles
Authors:
Nathan F. Lepora
Abstract:
Using the embedded defect method, we classify the possible embeddings of a 't Hooft-Polyakov monopole in a general gauge theory. We then discuss some similarities with embedded vortices and relate our results to fundamental monopoles.
Using the embedded defect method, we classify the possible embeddings of a 't Hooft-Polyakov monopole in a general gauge theory. We then discuss some similarities with embedded vortices and relate our results to fundamental monopoles.
△ Less
Submitted 23 July, 2002; v1 submitted 27 June, 2001;
originally announced June 2001.
-
Dualising the Dual Standard Model
Authors:
Nathan F. Lepora
Abstract:
We discuss how the dual standard model and the dualised standard model are complementary theories. That is, how their implications have no overlap, whilst together they explain most features of the standard model. To illustrate how these two theories might be combined we consider the dual standard model in a theta vacuum. Whilst there are issues to be considered, the dual standard model does the…
▽ More
We discuss how the dual standard model and the dualised standard model are complementary theories. That is, how their implications have no overlap, whilst together they explain most features of the standard model. To illustrate how these two theories might be combined we consider the dual standard model in a theta vacuum. Whilst there are issues to be considered, the dual standard model does then appear to become naturally dualised. This supports an origin of a dual formulation of the standard model through the properties of SU(5) solitons in a theta vacuum.
△ Less
Submitted 4 April, 2001; v1 submitted 7 February, 2001;
originally announced February 2001.