TacPAC is one of the most useful 2026 papers for anyone training VLA policies or action-chunking robot policies for real contact-rich manipulation. The core problem is familiar: a robot observes the scene, the model predicts a long action chunk, and the controller executes that chunk. Chunking makes motion smoother and reduces per-step inference pressure, but it has a sharp weakness once contact begins. During insertion, reorientation, delicate grasping, or tight assembly, the most important information is often no longer visible in RGB images. It is in tactile feedback.
The original paper is TacPAC: Tactile Prediction and Real-Time Action Correction in World-Action Models for Contact-Rich Manipulation, by Zipei Ma, Xiaofei Wei, Junzhe Jiang, Shunlin Lu, and Li Zhang, posted to arXiv on September 4, 2026. The official repository is LogosRoboticsGroup/TacPAC, released under the MIT License. It includes the model code, tactile preprocessing, training scripts, deployment protocol, and tests. The datasets and checkpoints used in the paper are still being prepared for public release, so this guide focuses on how to understand the method, install the repo, adapt your own LeRobot dataset, train the two stages, and run the stateful prepare/refine inference protocol.
If you are new to VLA policies, start with VLA models in robotics and Diffusion Policy action chunking. For sensor background, Tactile Sensing for Manipulation explains why touch becomes decisive after vision reaches its limit.
The Problem TacPAC Solves
A VLA or world-action model typically receives the current observation, a language instruction, and proprioception, then predicts an action sequence a_t ... a_{t+H-1}. TacPAC uses a horizon of H=48. Commands are issued at 30 Hz, so one chunk spans about 1.6 seconds of robot motion. That is long enough for many contact events to happen: a plug touches the edge of a socket, an expansion card misses a slot by a few millimeters, a bottle starts slipping in the gripper, or a potato chip begins to deform.
Vision is still necessary. It gives global scene context, object identity, and coarse alignment. But once the robot has made contact, RGB often cannot see pressure, micro-slip, surface deformation, or small hidden misalignments inside a slot. Tactile sensors can reveal these cues directly. The trouble is that simply feeding the current tactile image into a policy is not enough. The policy also needs to know what tactile signal the current plan expected to create.
TacPAC's key idea is to make tactile prediction actionable. During planning, the base world-action model predicts future tactile observations together with future visual observations and the action chunk. After the chunk is planned, TacPAC caches the predicted tactile contact and the action representation. During execution, a tactile expert reads every new tactile image against this cache. If the real tactile signal differs from the expected contact, the expert outputs a delta action for the part of the chunk that has not yet been executed.
That distinction matters. Tactile prediction is not just an auxiliary loss. It becomes the reference signal for online correction.
The Method in One Picture
TacPAC has three main components:
| Component | Role |
|---|---|
| Video expert | Predicts future visual and tactile observations in latent space |
| Action expert | Generates an action chunk through flow matching while attending to predictions |
| Tactile expert | Reads a new tactile image against the cache and corrects the action suffix |
The base model is a tactile-predictive world-action model. It does not only imagine future camera frames. It also predicts future tactile views. The action expert does not act independently; it generates its chunk while attending to these predicted observations. This means the action chunk is conditioned on an expected contact trajectory.
Once the chunk is generated, TacPAC performs a clean prefill pass and builds a layer-wise tactile-action KV cache. This cache stores both the expected tactile contact and the plan representation. While the robot executes the chunk, each new tactile frame is passed into the tactile expert. The expert does not regenerate the chunk. It performs one forward pass over the reusable cache and returns Delta a for only the unexecuted suffix. If the robot has already executed m steps, the correction is written only to steps m ... H-1.
For an intuitive mental model, think of TacPAC as making the plan state its tactile promise. The base model says, "If I push the plug along this trajectory, the tactile sensor should feel pattern A." During execution, the sensor sees pattern B. The tactile expert asks, "How does B differ from A, and how should the remaining actions change?" This makes feedback contextual instead of isolated.
Why Tactile Prediction Alone Is Not Enough
The paper makes a subtle but important point: predicting future tactile observations by itself is useful, but it does not unlock the full gain. In the component ablation, adding tactile prediction without the tactile expert raises average success from 22% to 37%. That is an improvement over the vision-only base model, but it is far below the full TacPAC result of 64%.
The reason is a timing mismatch. Future tactile prediction is made before execution. Real tactile feedback arrives during execution. If the prediction only influences the initially generated chunk, the model cannot react when the object is slightly misaligned, friction differs from training, contact arrives early, or deformation evolves unexpectedly. TacPAC turns prediction into a runtime reference, so late tactile feedback can still modify the plan.
This is an important design lesson for VLA stacks. Auxiliary prediction can improve representations, but real robots need an execution-time path through which predictions affect control. Otherwise the model may "know" that contact should happen, while having no mechanism to update the active action sequence when the actual contact is different.
Architecture Details
TacPAC is built around a Mixture-of-Transformers design. In the paper configuration, the video, action, and tactile experts share 30 Transformer layers and identical attention geometry: 24 heads with head dimension 128. The video expert uses hidden/feed-forward widths of 3072/14336, while the action expert and tactile expert use 1024/4096. A linear bridge maps the video frontend width from 3072 to 1024 for the tactile expert.
Each action step consists of a seven-dimensional joint command plus a one-dimensional gripper command. The joint command is expressed relative to the joint configuration at the first step of the chunk. The Flexiv setup uses two RGB views and two tactile views:
observation.images.third_view
observation.images.left_wrist_view
observation.images.left_wrist_left_tactile
observation.images.left_wrist_right_tactile
Visual views are rendered at 256x256. Tactile views are rendered at 128x128 on the shared canvas. The video tokenizer uses a VAE temporal stride of 4 and a future-frame stride of 4, so one latent frame covers 16 consecutive actions. With H=48, one horizon spans three video latent frames.
The tactile expert is initialized from the trained action expert. That is a practical choice because both modules operate over compatible action representations and temporal structure. Its output head is zero-initialized, so the first correction is the identity. At the beginning of Stage 2, the expert does not destroy the base plan; it gradually learns residual corrections.
The cache is the runtime trick that makes the method fast. Without it, each tactile update would need to regenerate the action chunk or reprocess much of the plan. With the layer-wise cache, one tactile correction takes 30.4 ms in the paper's setup, or 32.9 Hz. Regenerating a chunk takes 628.6 ms, so correction is 20.7 times cheaper.
Installation
The repository expects Python 3.10 or newer and a CUDA-capable PyTorch environment. Important dependencies include transformers==4.57.0, accelerate==1.12.0, deepspeed==0.16.9, lerobot==0.3.4, pyrealsense2, msgpack, pyzmq, websockets, diffusers, timm, and opencv-python.
git clone [email protected]:LogosRoboticsGroup/TacPAC.git
cd TacPAC
conda create -n tacpac python=3.10 -y
conda activate tacpac
pip install -r requirements.txt
pip install flash-attn --no-build-isolation
pip install -e .
Next, download the Wan2.2 TI2V backbone and update local paths in starVLA/config/training/vla/starvla_wam.yaml. Check dit_path, vae_path, text_encoder_path, and tokenizer_path. If you only want to inspect code or run CPU unit tests, you may not need all model assets immediately. For training and inference, missing backbone paths will fail during model loading.
Because the paper's private datasets are not included in the repo yet, you need your own LeRobot-format demonstrations. This is not a minor detail. TacPAC depends heavily on alignment between RGB, tactile frames, proprioception, and actions. If timestamps are offset, the tactile expert will learn corrections for the wrong contact phase.
Preparing LeRobot Data
A minimal sample should contain:
observation.images.third_view
observation.images.left_wrist_view
observation.images.left_wrist_left_tactile
observation.images.left_wrist_right_tactile
observation.state
action
task
Register the local dataset and its video_keys in starVLA/dataloader/vla/mixtures.py. Keys that contain tactile are treated as tactile views. Stage 2 emits tactile_now and tactile_offset samples, which simulate the situation where a robot has already executed part of a chunk before receiving a fresh tactile frame.
Precompute text embeddings before training:
python scripts/vla/precompute_text_embeds.py \
--config_yaml starVLA/config/training/vla/starvla_wam.yaml \
--datasets.vla_data.data_mix flexiv_plug_4views \
--datasets.vla_data.text_embedding_cache_dir data/text_embeds_cache/flexiv_plug_4views
If you do not have a tactile sensor, do not simply replace tactile with cropped RGB and expect paper-like behavior. TacPAC is designed for tactile images from sensors such as the InTac S1 used in the paper. You can adapt it to another tactile sensor or simulator, but you must keep the representation consistent between training and inference. The repo's shared preprocessing module, starVLA/dataloader/vla/tactile_stress.py, is there to prevent a common failure mode: training with one tactile normalization pipeline and serving with another.
Stage 1 Training: Tactile-Predictive WAM
Stage 1 trains the base model. The video expert predicts future visual and tactile observations. The action expert denoises the action chunk. This is the expensive stage because the model is learning both scene/contact dynamics and the mapping to actions.
NPROC_PER_NODE=8 bash scripts/vla/train_WanMoTJoint.sh \
flexiv_plug_4views \
stage1
The paper trains both stages on one node with eight NVIDIA H100 GPUs, global batch size 64, for 10 epochs. Smaller labs can start with a narrow task, fewer views, smaller batch size, and gradient accumulation. The first goal is not to reproduce the final number. It is to make the closed-loop contract work: the dataset loads correctly, action dimensions match the robot, tactile views are synchronized, loss decreases, and rollouts do not drift because of frame misalignment.
When debugging Stage 1, visualize three things: predicted future tactile frames, the action chunk, and real tactile frames from rollout. If the tactile prediction is constant or blurry, the model has not learned contact dynamics. If the action chunk is smooth but tactile prediction does not align with the physical contact phase, Stage 2 will not have a useful reference.
Stage 2 Training: Tactile Expert
Stage 2 is what turns TacPAC from a tactile-predictive model into a tactile-corrective controller. The base model, VAE, text encoder, and proprioceptive encoder are frozen. Only the tactile expert is trainable. The paper uses a base learning rate of 1e-4 with a cosine schedule.
NPROC_PER_NODE=8 bash scripts/vla/train_WanMoTJoint-TacExpert.sh \
flexiv_plug_4views \
results/Checkpoints/vla/<stage1-run>/final_model/pytorch_model.pt
Each frozen plan and prefill cache is shared by K=4 correction offsets. In plain language, one generated chunk becomes several training questions: "If the robot has already reached this offset and the current tactile image looks like this, how should the remaining actions change?" The loss should apply to the suffix that can still be changed, not to the prefix that has already been executed.
When porting TacPAC to a different robot, check three interfaces carefully. First, action dimension: the paper uses seven joint commands plus a gripper command, while your robot may be a 6-DoF arm, dual-arm system, or mobile manipulator. Second, tactile geometry: each sensor has different resolution, noise, lighting, elastomer behavior, and contact patterns. Third, control frequency: if you do not run at 30 Hz, then a 48-step horizon no longer equals 1.6 seconds, and correction timing must be adjusted.
Inference: Prepare and Refine
TacPAC's deployment server exposes a stateful tactile protocol:
CKPT=results/Checkpoints/vla/<tacpac-run>/final_model/pytorch_model.pt \
PORT=5556 \
bash deployment/local_infer-wan-tac.sh
The client calls prepare to generate a new action chunk and receive a plan_id. The server also builds the tactile-action cache for that plan. While the robot executes the chunk, the client repeatedly calls refine with the latest tactile views, execution offset, and executed action prefix. The server returns a corrected suffix for the same plan.
A simplified control loop looks like this:
plan = client.prepare(obs, instruction)
actions = plan["actions"]
plan_id = plan["plan_id"]
for m in range(len(actions)):
execute(actions[m])
if new_tactile_frame_ready():
update = client.refine(
plan_id=plan_id,
tactile=read_tactile_views(),
offset=m + 1,
executed_prefix=actions[: m + 1],
)
actions[m + 1 :] = update["corrected_suffix"]
The production rule is simple: never overwrite actions that have already passed their deadline. TacPAC is asynchronous. The robot keeps executing the latest committed chunk, while corrections are written only to the unexecuted suffix. Because one tactile expert pass takes 30.4 ms, most corrections arrive before the next control step. The paper reports that only 1.6% of tactile-expert calls overlap with the robot advancing by one or more control steps during computation.
Results
TacPAC is evaluated on five real-robot tasks: charger-plug insertion, multi-object fruit transfer, fragile potato-chip transfer, empty-bottle uprighting, and expansion-card insertion. Each method is tested for 20 trials per task. The physical setup uses a Flexiv Rizon 4 robot, a wrist-mounted Intel RealSense D405, a third-person Intel RealSense D435i, and a gripper with two InTac S1 tactile sensors.
| Variant | Plug | Fruit | Chip | Bottle | Card | Avg. |
|---|---|---|---|---|---|---|
| Vision only | 15 | 30 | 60 | 5 | 0 | 22 |
| Without tactile expert | 35 | 50 | 65 | 20 | 15 | 37 |
| Without tactile prediction | 40 | 25 | 45 | 30 | 25 | 33 |
| Without tactile cache | 40 | 50 | 75 | 40 | 30 | 47 |
| TacPAC | 80 | 65 | 90 | 40 | 45 | 64 |
The external baselines include pi0.5, ACT, VITaL, LingBot-VA, Dream-Tac, and T-Rex. TacPAC achieves the highest success rate on all five tasks. It improves the average from 22% for the vision-only base model to 64%, and beats the strongest evaluated baseline by 16 percentage points. The tightest tasks, expansion-card insertion and bottle uprighting, show why online tactile correction matters: very small physical deviations can decide success, and those deviations are often hard or impossible to see.
Practical Lab Checklist
If you want to try TacPAC in a smaller lab, keep the first experiment narrow:
- Choose one contact-heavy task, such as connector insertion, slotting, or lifting a slippery object.
- Mount tactile sensors firmly on the gripper and log raw tactile images with timestamps.
- Collect a LeRobot dataset with RGB views, left/right tactile views, proprioception, actions, and task text.
- Replay the dataset and verify alignment: tactile activity should spike exactly when contact occurs.
- Train Stage 1 on one task and inspect whether predicted tactile frames follow the contact phase.
- Train Stage 2 and confirm that suffix corrections do not introduce jerky motion.
- Run inference slowly first, then increase toward the real control rate.
TacPAC is not "add touch to a VLA and hope." Its value is the explicit contract among the planned chunk, the predicted contact, the cache, and the runtime correction path. If that contract is wrong, the model will correct the wrong thing. If it is right, tactile feedback becomes contextual, fast, and directly useful for correcting an active action chunk.
Takeaway
TacPAC moves tactile prediction from representation learning into control. Instead of predicting future touch and leaving it unused during execution, it caches the tactile expectation of the plan and uses a tactile expert to correct the unexecuted action suffix. For contact-rich manipulation, that is a practical division of labor: vision plans globally, the world-action model imagines contact, tactile sensors verify what actually happened, and correction is much cheaper than replanning.
For teams training VLA policies on LeRobot or deploying action-chunk policies outside the lab, the design pattern is worth remembering: do not only ask the model "what action should come next?" Also ask, "what contact did this plan expect, and how is the real tactile signal deviating from that expectation?"
