内容摘要
Does VTLA have to be trained from scratch? Combining touch with a state-of-the-art (SOTA) VLA model is a better approa…
Does VTLA have to be trained from scratch? Combining touch with a state-of-the-art (SOTA) VLA model is a better approach.
Recently, Xense Robotics, together with the University of Oxford, the University of California San Diego, Shanghai Jiao Tong University and Sun Yat-sen University, officially released two frontier research results in tactile sensing — TouchGuide and TacUMI. The papers are now available on arXiv, with project pages and code to be released progressively.
Vision tells us "looks good"; only touch can guarantee "really good."
In fine contact-rich manipulation, vision alone can only tell us which actions "look feasible," but true success often depends on the combined effect of vision and in-hand information (i.e., the contact information within the gripper provided by touch). TouchGuide's core idea is to give the policy access to touch at inference time, fusing what is "seen" with what is "felt" into the action space, and thereby pushing the action distribution toward real contact physics.
TouchGuide does not retrain a new policy from scratch; instead, it provides "tactile guidance" to an existing vision-based policy at inference time. In terms of workflow, it first uses vision to generate a coarse action, then a task-relevant Contact Physics Model (CPM) produces a feasibility score that corrects and guides the action during sampling. The CPM is trained via contrastive learning on a small number of expert demonstrations, outputting a feasibility signal of "whether the action conforms to contact physics." As a result, it can be applied to diffusion policies and seamlessly integrated into flow-matching policies, enabling cross-policy tactile fusion.
To validate this inference-time tactile guidance paradigm, the team designed five representative contact-intensive tasks: shoe-lacing (long-horizon, fine-grained task), chip transfer (dynamic interaction with fragile objects), cucumber peeling (continuous contact with dual-arm coordination), vase wiping (contact-dense manipulation), and lock picking under a low-data regime (only 20 demonstrations, requiring high precision). These tasks cover key challenges including complex contact, long horizons, force control with fragile objects, fine manipulation and low-demonstration learning.
The results show that TouchGuide delivers consistent improvements across different base policies and different tactile modalities: for example, on π0.5, TouchGuide with tactile images achieves significantly higher average success rates than the vision-only baseline; it also brings clear gains on DP. More importantly, it reduces failures caused by "first contact," pulling actions back from regions that are visually plausible but physically infeasible in contact, and maintains strong generalization across different robots and different tactile modalities.
High-quality tactile guidance presupposes multimodal data containing high-quality touch signals, so the team also released TacUMI: a low-cost, high-precision handheld multimodal tactile data collection system. TacUMI uses Vive Tracker and Lighthouse positioning to obtain accurate poses, delivers direct tactile feedback through a rigid fingertip interface, and synchronizes vision, touch and pose in a unified pipeline. Meanwhile, it keeps structural consistency between the collection end and the execution end, and can be quickly attached to different robot arms via adapters, closing the loop between "collectable data" and "usable policies."
On the hardware design front, TacUMI is built around "direct tactile feedback": a rigid fingertip interface, Xense tactile sensors, magnetic encoders, a wrist camera and an onboard end-effector computing unit provide synchronized capture of high-resolution tactile images and force-field estimation. This minimal structure with unified interfaces makes high-precision tactile data acquisition lighter and more scalable.
For more details, please follow:
Project Page: https://martelzhang.github.io/touchguide/
GitHub: https://github.com/martelzhang/TouchGuide
arXiv: https://arxiv.org/abs/2601.20239
Source: https://mp.weixin.qq.com/s/XBequSE2rcLLOg2ALlYNwg
