T-Rex: Fast Tactile-Reactive VLA for Dexterous Manipulation with LeRobot v3.0

Vision tells a robot where the egg is, where the page edge starts, and where the light bulb should go. Touch tells it whether the egg is about to crack, whether the fingertip is slipping off the paper, and whether the bulb thread is engaging correctly. That difference is the motivation behind T-Rex: Tactile-Reactive Dexterous Manipulation, a new robotics paper and open-source project from UC Berkeley, NVIDIA, Stanford, and collaborators.

The official project page is tactile-reactive-dexterous.github.io, the paper is arXiv 2606.17055, the code is ZhuoyangLiu2005/T-Rex, and the public dataset is released on Hugging Face as zekaiwang/trex_dataset. The key idea is not merely "add tactile tokens to a VLA." T-Rex redesigns the pipeline around the fact that touch is naturally high-frequency while vision-language planning is slower. It uses one clock for visuomotor planning and another faster clock for tactile refinement.

For practitioners, T-Rex is useful because the public repo ships post-training and inference code, released pretrain/midtrain checkpoints, and a LeRobot v3.0 data path. You do not need to reproduce the full 22,889-hour human egocentric pretraining stage or the complete internal mid-training pipeline before doing a meaningful experiment. A realistic path is: start from the released midtrain checkpoint, collect task-specific demonstrations on compatible tactile hardware, convert the data to JSON or LeRobot v3.0, post-train, and serve the policy through the slow/fast inference protocol.

What Problem T-Rex Solves

Most current Vision-Language-Action systems are visually competent but tactilely slow or blind. They can parse a natural language instruction, localize objects, and produce an action chunk. But when the fingertips make contact, the key signal often arrives through force, torque, shear, and local deformation. Vision may not resolve a 0.5 mm slip, a tiny contact edge, or a force threshold that decides whether the task succeeds.

T-Rex focuses on tactile-reactive dexterous manipulation: manipulation where the robot needs closed-loop responses to touch while the action is still unfolding. Examples include flipping a page, transferring an egg, wiping a plate, applying toothpaste, opening a lock, extracting a card, dealing poker cards, and screwing a light bulb. These tasks are difficult because success depends on delicate force control, deformable objects, small clearances, or bimanual coordination.

The paper addresses three bottlenecks:

1. Data scarcity: large tactile-synchronized dexterous datasets are rare, especially for bimanual hands rather than simple parallel grippers.
2. Frequency mismatch: VLA backbones run at relatively low rates, while tactile corrections need to happen faster.
3. Static tactile encoding: a single tactile frame misses the temporal structure of slip, pressure change, and deformation.

T-Rex responds with a 100-hour tactile-rich dataset, a variable-rate Mixture-of-Transformers architecture, and a temporal tactile VQ-VAE encoder.

The Dataset: Tactile-Rich LeRobot v3.0

The T-Rex Dataset is collected on a bimanual Dexmate Vega-1 platform with two 7-DoF arms and two 22-DoF Sharpa Wave dexterous hands. The full benchmark platform has 58 DoF. Perception includes a head ZED X Mini stereo camera and two wide-view wrist cameras. Tactile data comes from ten fingertip sensors, five on each hand. The released data includes RGB streams, robot state, actions, raw tactile images, deformation maps, and estimated 6D wrenches.

The dataset design is worth studying. Instead of recording a small number of long tasks, the authors organize teleoperation around object × motor primitive combinations. The project page describes 207 household objects and 22 motor primitives. After removing physically infeasible pairs, the data covers 502 meaningful combinations with roughly 17 demonstrations each. This creates broad coverage of reusable contact behaviors such as pressing, sliding, rotating, inserting, wiping, folding, and pinching.

The public release is approximately 50 hours and more than 5,400 trajectories in LeRobotDataset v3.0 format. The README lists the important schema:

The LeRobot path also writes a meta/trex_norm_stats.json sidecar with quantile normalization and tracking error statistics. This matters because tactile policies are sensitive to normalization. If the train-time loader and deploy-time preprocessing disagree, the fast tactile expert may react to the wrong force scale.

If you are new to the ecosystem, read LeRobot hands-on first. T-Rex is a more advanced version of the same data discipline: synchronize multiple cameras, proprioception, action chunks, and high-rate tactile sequences under a standard dataset format.

Architecture: Two Clocks Inside One VLA

T-Rex uses a Qwen3-VL-2B backbone and implements a variable-rate Mixture-of-Transformer-Experts (MoT). The model separates control into three expert streams:

• Latent expert: keeps visual-language context and predicts future latent representations, preserving task-level reasoning.
• Action expert: plans a coarse action chunk from vision, language, state, and latent context. This is the slower visuomotor planning path, around 5 Hz in the public description.
• Tactile expert: receives fresh tactile information and refines the remaining action through residual corrections. This is the faster path, around 20 Hz.

The central mechanism is asynchronous cascaded flow matching. In simple terms:

At the start of an action chunk:
  camera + language + state -> slow pass
  action expert runs partial denoising
  server caches KV state and an intermediate x_split

During the chunk:
  new tactile F6/deform -> fast pass
  tactile expert continues from x_split
  server returns the refined action

This avoids rerunning the full vision-language stack every time touch changes. The slow path provides context; the fast path reacts to contact. That is the design choice that makes T-Rex different from simply concatenating tactile features to VLA inputs.

For robotics engineers, this is the most reusable lesson from the paper. Multimodal robot policies should respect the control frequency of each modality. Vision is excellent for semantics and coarse geometry. Tactile sensing is excellent for local closed-loop correction. Proprioception anchors the robot state. Treating all of them as one synchronous token soup is convenient, but often wrong for real-time contact.

Temporal Tactile VQ-VAE

Tactile information is temporal. A single force vector is much less informative than the pattern of force growth, shear reversal, pressure release, and fingertip deformation over the last few frames. T-Rex uses a temporal tactile VQ-VAE over a rolling F6 history window.

According to the README, the default training path encodes tactile codes on the fly through an embedded VQ-VAE. The trainer runs with --use_tactile_vqvae 1, so no offline tactile-code baking is required. The raw input history has shape [B, window, 10, 6]: batch, time window, ten fingertips, and six wrench channels per fingertip. The released midtrain checkpoint already embeds the VQ-VAE, and train.py / test.py can auto-detect the required settings from training_args.json.

This is a practical deployment advantage. You do not need to maintain a separate tactile encoder service or worry about a mismatch between offline codes and online force history. The model receives raw F6 history, normalizes it, tokenizes it internally, and uses those discrete tactile tokens for fast refinement.

Installation

The main branch of the repo ships post-training and inference code. The full pretraining/mid-training pipeline lives in the full-pipeline branch, and the full pretrain/midtrain corpora are not part of the main release. Beginners should start from the released midtrain checkpoint.

Recommended environment:

• Linux workstation or GPU server.
• Python 3.10.
• NVIDIA GPU with a CUDA setup matching the PyTorch wheel.
• Enough disk for video-heavy LeRobot datasets.
• Compatible robot logs if you plan to fine-tune on your own task.

Basic install:

conda create -n trex python=3.10 -y
conda activate trex

pip install torch==2.6.0 torchvision==0.21.0 \
  --index-url https://download.pytorch.org/whl/cu124

git clone https://github.com/ZhuoyangLiu2005/T-Rex.git
cd T-Rex
pip install -e .

Install LeRobot only if you use the LeRobot v3.0 data path:

git clone https://github.com/huggingface/lerobot.git
pip install -e /path/to/lerobot

Every .sh script in T-Rex has an editable header. Open it before running. Set PROJECT_ROOT, conda environment path, dataset path, checkpoint path, and other local variables. Do not assume the default paths match your machine.

Preparing Your Own Task Data

T-Rex supports two post-training data paths.

The first is the default JSON path. Raw task episodes are expected under directories such as:

<root>/success/episode_0001/
  episode.h5
  head.mp4
  wrist_left.mp4
  wrist_right.mp4

You can edit utils/gen_json_bimanual.sh or call the converter directly:

python utils/gen_json_tac_deltabase_eef_bimanual_parallel.py \
  --data_roots /path/to/raw/task_a /path/to/raw/task_b \
  --img_save_root /path/to/training_data/images \
  --json_save_root /path/to/training_data/json \
  --task_name place_card_lr_bimanual_stride1 \
  --json_name_base place_card_deltabase_axis_eef_lr_bimanual_stride1_train \
  --instruction "Pick up the card and insert it into the slot" \
  --num_workers 16

The second is the LeRobot v3.0 path. Edit DATA_ROOTS, OUTPUT_ROOT, REPO_ID, and LEROBOT_SRC inside utils/convert_inlab_to_lerobot.sh, then run:

bash utils/convert_inlab_to_lerobot.sh

For LeRobot training, set these at the top of scripts/train.sh:

DATA_FORMAT="lerobot"
LEROBOT_ROOT="/data/lerobot/my_trex_task"

The model, cascaded-flow loss, and training loop remain unchanged. The LeRobot dataloader emits the same batch dictionary as the JSON dataset, and the embedded VQ-VAE tokenizes raw F6 history from LeRobot delta_timestamps.

Before launching a long training run, inspect one batch:

• Head, left wrist, and right wrist videos are mapped correctly.
• observation.tactile_f6 has ten fingertips and six channels per finger.
• The action dimension matches the expected 62.
• Instructions are consistent and not overly vague.
• Normalization stats are generated after conversion.
• A single batch can run through the model without shape mismatch.

If your robot has a standard arm and a simple gripper with no comparable tactile sensors, do not expect the released T-Rex checkpoint to deploy directly. In that case, use the repo as an architecture reference and data-format guide rather than a drop-in policy.

Post-Training from the Midtrain Checkpoint

The full T-Rex training recipe has three stages:

1. Human egocentric pre-training: latent and action experts learn broad visual-language and visuomotor priors from 22,889 hours of first-person human videos.
2. Tactile-grounded robot mid-training: 100 hours of synchronized bimanual tactile robot data adapts those priors to real contact dynamics and trains the tactile expert.
3. Skill-specific post-training: the policy is fine-tuned on roughly 100 demonstrations for a downstream task.

The public main branch is designed for stage 3. The model zoo includes miniFranka/T-Rex_pretrain_mecka22k_epoch1 and miniFranka/T-Rex_midtrain_mecka23k_ucb100_vqvae_epoch6. For your own task, resume from the midtrain checkpoint:

# Edit scripts/train.sh
RESUME_CHECKPOINT="/path/to/T-Rex_midtrain_mecka23k_ucb100_vqvae_epoch6"
RESUME_SOURCE="midtrain"
DATA_FORMAT="lerobot"   # or json
LEROBOT_ROOT="/data/lerobot/my_task"

bash scripts/train.sh

Multi-node training is supported through MASTER_ADDR, MASTER_PORT, NUM_MACHINES, and MACHINE_RANK. The effective batch size is train_bsz_per_gpu × NUM_PROCESSES × gradient_accumulation_steps. Start single-node first unless you already know your data pipeline is stable. Distributed training magnifies small path, timestamp, and normalization mistakes.

Inference: Slow/Fast ZMQ Protocol

Inference is served by scripts/test.py through scripts/test.sh. The server uses a ZMQ REP socket with three request modes:

Set MODEL_PATH in scripts/test.sh:

MODEL_PATH="/path/to/posttrained/checkpoint"
bash scripts/test.sh

The robot-side client is hardware_code/eval/eval_trex_async.py. It sends a slow request at the beginning of a chunk, then sends fast requests as new tactile packets arrive. For checkpoints with an embedded VQ-VAE, the server keeps a rolling 16-frame F6 buffer and performs tactile tokenization internally. For a clean no-tactile ablation, --disable_tactile 1 makes the policy run through the action expert without the tactile expert.

In deployment, measure the complete latency path: camera frame time, tactile packet time, slow request time, fast request time, and command time at the low-level controller. A fast tactile expert is only useful if the tactile signal is fresh. A 30-50 ms timestamp error can teach the policy the wrong causal relationship between action and contact.

Benchmark Results

T-Rex is evaluated on 12 real-world tactile-reactive tasks. The task set spans force-sensitive contact, deformable-object manipulation, and bimanual force-deformation coordination. The project page reports 16 randomized trials per task. Macro-average success rates are:

The headline is that T-Rex beats the strongest baseline by more than 30 absolute points. The more interesting lesson is that π0.5 + tactile performs worse than plain π0.5. This supports the paper's argument: tactile information is not automatically beneficial. If the architecture and training recipe do not respect the frequency and dynamics of touch, tactile signals can confuse a pretrained VLA.

Some per-task results are especially strong: Flip Page reaches 96%, Transfer Egg 75%, Wipe Plate 69%, Split Cup 78%, Acid-Base Neutralization 76%, and Extract Card 70%. Harder long-horizon, tight-tolerance tasks remain challenging. Screw Lightbulb is reported at 35%, which is still far from solved but meaningful for real bimanual dexterous manipulation.

The training recipe ablation is also important. Human pretraining provides broad semantic and visuomotor priors. Tactile-grounded mid-training bridges those priors to robot-executable contact dynamics. Post-training alone on around 100 demonstrations is not enough to teach robust tactile reflexes from scratch.

How T-Rex Relates to Other Tactile VLA Work

If you have read FM-VLA, that project uses force history as memory tokens for non-Markovian contact tasks. If you have read TORL-VLA, that approach combines tactile sensing with online RL to refine a policy during deployment. T-Rex is different: it builds a foundation-style tactile-reactive architecture with variable-rate experts and a large tactile mid-training dataset.

Quick comparison:

For readers learning VLA models, the general lesson is that multimodal robotics is not just about adding more inputs. Each modality has its own frequency, noise profile, and control role. Good architecture respects those differences.

A Practical Roadmap for Small Labs

Most small labs will not have the exact Vega-1 plus Sharpa Wave setup. You can still apply the T-Rex recipe in stages:

1. Master LeRobot v3.0 first: collect camera, state, and action data on a simpler robot; verify replay and dataloader behavior.
2. Add a small tactile signal: wrist force/torque or a simple fingertip sensor is enough to debug synchronization and normalization.
3. Collect short motor primitives: press, slide, rotate, insert, wipe, and pinch are easier to scale than full long-horizon tasks.
4. Measure latency early: tactile is valuable only if the fast loop is truly faster than rerunning the full vision stack.
5. Fine-tune carefully: use the released checkpoint when embodiment and sensor layout are close enough; otherwise treat T-Rex as a blueprint.

The most common mistake is focusing on model size before the data pipeline is clean. With tactile learning, timestamp errors are especially damaging. Plot action commands, joint response, tactile F6, and video frames around a contact event. If the event does not line up, fix synchronization before training.

Conclusion

T-Rex is important because it brings tactile feedback into VLA control at the right temporal scale. A slow vision-language-action path plans the chunk; a fast tactile expert refines the action inside the chunk; a temporal VQ-VAE converts high-frequency fingertip force history into stable tokens. The public release gives the community a strong reference for tactile-rich LeRobot v3.0 datasets, post-training, and slow/fast inference.

For beginners, do not treat T-Rex as a universal plug-and-play policy. Treat it as a blueprint for contact-rich robot learning: collect reusable primitives, standardize data with LeRobot, preserve tactile timing, post-train from a compatible midtrain checkpoint, and measure real-time latency. When these pieces are correct, touch stops being an auxiliary sensor and becomes the reflex layer that makes dexterous VLA manipulation work in the real world.

Related Posts

• LeRobot hands-on: from dataset to policy
• FM-VLA: Force Memory Tokens for contact-rich manipulation
• TORL-VLA: fine-tune VLA with tactile sensing and Online RL