TacInno

Teaching Vision-Language-Action Models the Contact Dynamics Beyond Vision and Action

*Equal contribution   †Corresponding author: zhibo.pang@pku.edu.cn

Paper Code BibTeX
Overview of TacInno. The tactile change mixes a component explained by images and actions with a contact response that only tactile input reveals. The TacInno tokenizer encodes the contact response into a code used only as a training target.
Overview. (a) The tactile change mixes a component explained by vision and executed actions with a contact response that only tactile input reveals. (b) The TacInno tokenizer encodes the contact response into the tactile innovation code. (c) Existing visuo-tactile VLA models versus TacInno, which uses the code solely as a training target. (d) Average and lowest per-task success rates on six UniVTAC tasks. (e) Success rates on four real-world tasks.

Abstract

Vision-Language-Action (VLA) models have recently been equipped with tactile sensing to perceive visually subtle contact events in contact-rich manipulation. However, adding tactile input yields inconsistent gains, and some visuo-tactile VLA models rely little on tactile input, because much of the tactile change, such as the gel deformation during gripper closure, is predictable from visual observations and executed actions. To address this problem, we introduce TacInno, which isolates the contact dynamics that vision and executed actions cannot explain, termed tactile innovation, and uses tactile innovation only as a training target. Specifically, the TacInno tokenizer, conditioned on visual observations and executed actions, compresses the future tactile change into a discrete code, and the policy, TacInno-VLA, learns to predict the code, adding no module for the code at deployment. TacInno-VLA achieves average success rates of 87.5% across six simulation tasks and 70.5% across four real-world tasks, outperforming the strongest visuo-tactile baseline by 10.7 and 9.0 percentage points, respectively. Ablation studies validate the contributions of visual and action conditioning, multi-frame encoding, and auxiliary code prediction. Together, these results demonstrate the effectiveness of tactile innovation as a supervision signal for contact-rich manipulation.

The idea

What can tactile sensing still tell a policy that already sees the scene and knows its own actions?

A tactile sensor measures how the contact responds to the robot's own actions. When the gripper closes, the gel deforms in a way the cameras and the executed actions already predict. What they cannot predict is how the contact responds, such as an object that starts to slip or a plug that jams.

Future tactile changeΔX
Predictable from vision and actionswhat the images and the executed actions already explain, such as the gel deformation during gripper closure
+
Tactile innovationwhat only tactile input reveals, such as incipient slip of a grasped object or a jamming plug

Feeding tactile observations to the policy leaves it to the action loss to find this residual, and predicting all future tactile signals spends most of the supervision on the predictable part. TacInno isolates the residual, borrowing the notion of innovation from estimation theory, and supervises the policy with it.

Try it

Stand a lying can upright by letting it turn in the fingertips

The can hides its mass and surface friction. Hold it firmly and lift it, then loosen the grip slowly: it turns upright in the fingertips and settles on the table.

  1. 1Lift, then loosen slowly
  2. 2Remove the conditioning
  3. 3Change the can

Hidden mass and friction ? · ?
Tactile change
Predicted from images + actions
Tactile innovation

Tactile innovation over timeinnovation this run –
|ΔX| (normalized)share that is tactile innovationcan tilt

acontactbliftcstarts to turnduprightereleased

A toy contact model that runs in your browser, not the trained TacInno tokenizer. The predictor receives your actions, sees the gripper, and sees the can once its pose is off by more than 6 degrees or 5 mm. It assumes an average mass and friction.

01 · conditional

Subtract what vision and actions explain

The tokenizer gives the visual observations and the executed actions to both its encoder and its decoder, so the code only has to carry what the two conditions cannot explain.

02 · dynamic

Encode a short window of contact

Transient events such as incipient slip span several control steps. The encoder reads four tactile frames from t to t+15 with causal temporal attention and outputs a 16-bit code.

03 · in the future

Use the code only as a training target

The policy predicts the code from current observations through four code queries that no other token attends to. The tokenizer and both auxiliary heads are removed at deployment.

Method

TacInno tokenizer

The encoder and the decoder are spatial-temporal Transformers whose frames hold tactile tokens, frozen DINOv2 image features, and executed actions. The encoder compresses the tactile observations of the window into four code tokens with 16 values each. The decoder reconstructs the future tactile change from the current tactile observation and the code. Because the decoder already receives the images and the actions, information about the tactile innovation reaches it only through the code.

Architecture of the TacInno tokenizer
TacInno tokenizer. The encoder compresses the tactile observations from t to t+15 into the code, and the decoder reconstructs the future tactile change. Both are conditioned on visual observations and executed actions, so the code retains the tactile innovation.

TacInno-VLA

TacInno-VLA is built on the pretrained VLA π0.5. Proprioceptive tokens, 60 tactile tokens from the three most recent tactile observations, and four learnable code queries are inserted into the action expert. A block-wise attention mask lets the code queries read the prefix, proprioceptive, and tactile tokens, while no other token reads the code queries. The code prediction loss therefore shapes the tactile representations that action prediction uses, rather than handing the code to the actions.

A contact-state head on the tactile encoder predicts contact properties such as contact area, shear, and indentation depth. At deployment the model is π0.5 with a 60-token tactile branch. No tactile pretraining is used.

Architecture of TacInno-VLA
TacInno-VLA. Two auxiliary heads predict the tactile innovation code and the current contact state during training and are removed at deployment.

Results

Six contact-rich tasks in UniVTAC

All policies are trained on the same 50 demonstrations per task and evaluated on the same 100 seeds per task. TacInno-VLA requires no tactile pretraining, whereas FTP-1 and N0-VTLA start from policies pretrained on external visuo-tactile data.

87.5%
average success on six simulation tasks
+10.7 points over the strongest visuo-tactile baseline
70.5%
average success on four real-world tasks
+9.0 points over the strongest visuo-tactile baseline
88% → 0%
on Lift Can when tactile input is removed
the actions of TacInno-VLA depend on tactile input
MethodTactileInsert HoleInsert TubeLift BottleLift CanPull-out KeyPut BottleAvg.
UniVTAC-ACTinput46554927412540.5
π0.5none88702625364047.5
FTP-π0.5input77994533198860.2
FTP-1input63996194408874.2
N0-VTLAinput99902481778175.3
SimpleTouchinput + predict94938265547376.8
TacInno-VLA (ours)input + innovation target95957888947587.5

Closed-loop success rates (%). Bold and underlined values denote the best and second-best results.

Closed-loop rollouts of TacInno-VLA

Insert Hole
Insert Tube
Lift Bottle
Lift Can
Pull-out Key
Put Bottle in Shelf

Each clip shows the head camera and the two tactile sensors, played at 2× speed.

Analysis

Does the policy use tactile input?

A higher success rate does not by itself show that a policy relies on tactile input. In the tactile-off test, the tactile input is replaced with a fixed no-contact frame at test time while all other inputs stay unchanged. On Lift Can, where the can must be held firmly enough to lift yet loosely enough to rotate in the grasp, TacInno-VLA drops from 88% to 0%. It reaches the can but does not lift it. The baselines retain most of their success rates.

PolicyWith tactile inputTactile-offChange
N0-VTLA8177−4
FTP-19480−14
SimpleTouch6555−10
UniVTAC-ACT2724−3
TacInno-VLA (ours)880−88

Lift Can, success rates (%) on the same 100 seeds.

Same seed, same policy. With tactile input (left), TacInno-VLA lifts the can. With a fixed no-contact tactile frame (right), it reaches the can but does not lift it.

Where does tactile innovation occur?

Temporal distribution of tactile innovation on a held-out Lift Can demonstration
Tactile innovation over time on a held-out Lift Can demonstration. The part of the tactile change explained by the code (orange) peaks before contact onset and at incipient slip, and vanishes once the motion of the can becomes visible to the cameras.

Ablation study

Each variant removes or alters one design of TacInno-VLA on the two lifting tasks. Removing the conditioning, so that the code encodes the entire future tactile change, lowers the average success rate by 21 points. Conditioning on vision alone, encoding only two frames, or letting the actions read the code queries also lowers it.

VariantLift CanLift BottleAvg.
TacInno-VLA (ours)887883.0
w/o conditioning636162.0
vision-only conditioning627769.5
two-frame code817276.5
w/o attention mask667068.0
w/o code loss846876.0

Lift Can and Lift Bottle, success rates (%) on 100 seeds per task.

Real world

Four contact-rich tasks on a Franka FR3

Wiping a board, inserting a plug into a power strip, lifting a lying can upright, and unscrewing a bottle cap, with a vision-based tactile sensor on the fingertip. Every policy is trained on the same 45 teleoperated demonstrations per task and evaluated for 50 trials per task. The code prediction loss improves every task by 16 to 26 percentage points.

Closed-loop rollouts of TacInno-VLA

Wipe Board
Insert Plug
Lift Can
Unscrew Cap

Recorded with an external camera, played at 4× speed.

PolicyWipe BoardInsert PlugLift CanUnscrew CapAvg.
FTP-18816666458.5
SimpleTouch8228627461.5
TacInno-VLA, tactile-off5210465240.0
w/o code loss6222505647.5
w/o tactile input6214527851.5
TacInno-VLA (ours)8838748270.5

Success rates (%) over 50 trials per task.

Real-world setup and the four tasks
Setup. A Franka FR3 arm with a parallel gripper, a vision-based tactile sensor on the fingertip, a third-person camera, and a wrist camera, with key frames of the four tasks.

Citation

BibTeX

@article{li2026tacinno,
  title   = {TacInno: Teaching Vision-Language-Action Models the
             Contact Dynamics Beyond Vision and Action},
  author  = {Li, Weile and Lou, Yunfan and Shi, Muqing and Zhang, Zihao and Huang, Zhenbiao and Yao, Boxian and Zhu, Xiyu and Pang, Zhibo},
  journal = {arXiv preprint arXiv:2611.26405},
  year    = {2026}
}