LaST-R1: Fine-tune VLA with Latent CoT and RL for 99.8%

Modern VLA (Vision-Language-Action) models — π0, GR00T-N1, OpenVLA — have demonstrated that combining vision and language can drive robots through complex manipulation tasks. But they share a fundamental weakness: they learn to imitate without truly understanding the physics behind the actions they take. And when you apply Reinforcement Learning (RL) on top, the RL only optimizes the action generation — leaving the internal reasoning process entirely untouched.

LaST-R1 was built to solve exactly this problem. By integrating Latent Chain-of-Thought (CoT) reasoning — hidden tokens that simulate physical reasoning — directly into the RL optimization loop, LaST-R1 achieves a 99.8% average success rate on the LIBERO benchmark using just 1 demonstration per task for the initial warmup stage.

Paper: LaST-R1: Reinforcing Action via Adaptive Physical Latent Reasoning for VLA Models — Hao Chen, Jiaming Liu, Zhonghao Yan, Nuowei Han, Renrui Zhang et al. (CUHK, Peking University, Simplexity Robotics), arXiv:2604.28192, 2026.

The Problem — Why Standard RL Falls Short

Picture teaching a robot to insert a hexagonal peg into a socket. With imitation learning, you show it 30 videos — the robot copies each motion. With plain RL, the robot trial-and-errors until it gets reward.

Both approaches miss the most critical step: physical modeling before action. No engineer inserts a peg without mentally noting: "this angle, this force, this approach direction." That sub-second internal reasoning is the foundation of precise execution.

With standard VLA + RL:

• RL only optimizes π(action | observation) — gradients never touch intermediate reasoning
• Latent representations (if any) are treated as a black box outside the optimization loop
• Result: policy works well in familiar conditions, but fails when object, background, or lighting changes

LaST-R1 changes this by making latent reasoning a first-class citizen in the RL loop.

Architecture — Three Interlocking Layers

LaST-R1 is built on three components, deliberately stacked on each other:

1. VLA Backbone — Qwen3-VL-4B + SigLIP2-Large

The foundation model is Qwen3-VL-4B (4 billion parameters) paired with SigLIP2-Large as visual encoder. This is a pragmatic choice: Qwen3-VL-4B has strong language understanding and is small enough for fine-tuning on accessible hardware, while SigLIP2-Large provides high-quality visual features for robotics tasks.

Input pipeline: RGB images from robot cameras + language instruction → tokenize → feed to LLM backbone.

2. DINOv3 Latent Representation — "Physical Memory"

This is LaST-R1's most distinctive component. Rather than letting the model learn latent representations from scratch, the authors use DINOv3 — a vision foundation model pre-trained to understand spatial structure and physical properties — to provide grounded latent targets.

How it works:

• DINOv3 processes the observation image and outputs a CLS token (dimension 4096)
• Top-k selection with k=2560 is applied to match the VLA embedding size
• These latent tokens represent the model's "physical understanding" of the current scene

Why DINOv3 outperforms alternatives:

DINOv3 preserves spatial structure — knowing where objects are in 3D space — without adding inference overhead.

3. Action Tokenizer — Parallel Decoding

LaST-R1 uses a parameter-free action tokenizer with parallel decoding at the output end. For single-arm robots (Franka Research 3): 7-DoF joint control. For dual-arm: 14-DoF through concatenation of both arms.

The full generation sequence is:

[Visual tokens] → [Latent CoT tokens] → [Action tokens]

The critical difference from other VLAs: latent CoT tokens are not frozen — they participate in RL optimization.

LAPO — The RL Algorithm That Reasons and Acts

LAPO (Latent-to-Action Policy Optimization) is the heart of LaST-R1. While standard PPO/GRPO only optimize the action distribution, LAPO jointly optimizes both the latent reasoning process and action generation.

How It Works

In each rollout step, LAPO:

1. Generates latent sequence autoregressively: z₁, z₂, ..., zₙ (n ≤ 8 tokens)
2. Generates action tokens conditioned on the generated latent sequence
3. Computes joint likelihood ratios for both latent tokens and action tokens

The key challenge: latent tokens are continuous (unlike discrete action tokens), so LAPO uses an isotropic Gaussian approximation to compute the likelihood ratio for latents, regularized via Euclidean distance:

Latent ratio ∝ exp(-∑ₖ ‖z_t,k^old − z_t,k^θ‖²)

Where z_t,k^old is the latent token from the old policy and z_t,k^θ from the current policy being optimized. This formula penalizes large deviations in latent reasoning — preventing policy collapse.

Unified loss function combining three components:

L_LAPO = L_action + α·L_latent + β·L_value

By optimizing all three simultaneously, when the model improves its value estimate (learns that a task is doable), gradients flow back through latent reasoning — forcing latent tokens to adapt and better support action generation.

Adaptive Latent CoT — Think as Much as Needed

Different tasks require different amounts of reasoning. A simple pick-and-place might need minimal thought; inserting a hexagonal peg while correcting for misalignment requires more. LaST-R1 handles this through Adaptive Latent CoT — automatic reasoning length adjustment.

The mechanism:

• The model can emit a <latent_end> token at M=4 candidate positions in the latent sequence
• When confident enough (p ≥ 0.99) that reasoning is complete → emit <latent_end> early → proceed to action generation
• During training: temperature-controlled exploration allows trying different reasoning lengths
• During inference: confidence-based strategy — terminate early if confident, continue otherwise

Ablation — Adaptive vs Fixed Reasoning Length:
├── Adaptive (M=4 candidate positions): 99.8%
├── Fixed length 8 tokens:              98.4%
├── Fixed length 4 tokens:              97.9%
└── Fixed length 1 token:               95.2%

Adaptive doesn't just hit the highest accuracy — it also saves compute by not wasting tokens on straightforward tasks.

Training Pipeline — Two Clear Stages

Stage 1: SFT Warmup — "Learning Basic Instincts"

Before RL, LaST-R1 needs a reasonable starting point. This SFT (Supervised Fine-Tuning) stage uses just 1 demo per task — an extremely small amount compared to most VLA fine-tuning pipelines.

Why only 1 demo suffices: DINOv3 already provides a strong physical prior; Qwen3-VL-4B already has strong language understanding from pretraining. Warmup only needs to "point the model in the right direction" — not teach it from scratch.

Pre-trained warmup models are uploaded to Hugging Face for all 4 LIBERO suites: Spatial, Object, Goal, and Long (LIBERO-10).

Stage 2: LAPO RL Post-Training — "Learning Through Practice"

After warmup, online RL with LAPO begins:

1. Rollout collection: Policy interacts with LIBERO simulator, collecting trajectories
2. Reward assignment: Binary reward (0/1) based on task completion
3. LAPO update: Compute joint loss, update both latent reasoning and action policy
4. Iterate: Repeat until convergence

The learning curves show LaST-R1 with LAPO converges significantly faster than Action-Only PPO — because latent reasoning gives gradients a clear direction instead of random exploration.

Installation

System Requirements

• Python 3.10+
• PyTorch ≥ 2.2.0
• CUDA ≥ 12.0
• GPU: A100/H100 recommended (RL training is VRAM-hungry); RTX 4090 works with smaller batch sizes

Step 1 — Clone and Create Environment

git clone https://github.com/CHEN-H01/LaST-R1.git
cd LaST-R1

conda create -n last-r1 python=3.10 -y
conda activate last-r1

Step 2 — Install PyTorch with CUDA 12.4

pip install torch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1 \
  --index-url https://download.pytorch.org/whl/cu124

Step 3 — Install Dependencies

pip install -r requirements.txt

Step 4 — Install veRL (RL Training Framework)

LaST-R1 uses veRL — ByteDance's distributed RL framework with FSDP/DeepSpeed support.

git clone -b v0.2.x https://github.com/volcengine/verl.git
cd verl
pip install -e .
cd ..

Step 5 — Install LIBERO Simulator

LIBERO is the standard manipulation benchmark with 4 task suites.

git clone https://github.com/Lifelong-Robot-Learning/LIBERO.git
cd LIBERO
pip install -e .
cd ..

Training

Download Warmup Checkpoint

Pre-trained warmup models are available on Hugging Face. For LIBERO-Spatial:

huggingface-cli download CHEN-H01/LaST-R1-warmup-spatial \
  --local-dir ./checkpoints/warmup_spatial

Also available: LaST-R1-warmup-object, LaST-R1-warmup-goal, LaST-R1-warmup-libero10.

Run RL Post-Training

The main script is at scripts/run_libero_rl_training.sh. Key parameters to configure:

#!/bin/bash

# Path to warmup checkpoint
SFT_MODEL_PATH="./checkpoints/warmup_spatial"

# Set to False for training (True for evaluation only)
VAL_ONLY=False

# Choose task suite
DATASET_NAME="libero_spatial"
# Options: libero_spatial | libero_object | libero_goal | libero_10

# Number of GPUs (adjust batch size accordingly)
NUM_GPUS=8

bash scripts/run_libero_rl_training.sh \
  --sft-model-path $SFT_MODEL_PATH \
  --val-only $VAL_ONLY \
  --dataset-name $DATASET_NAME \
  --num-gpus $NUM_GPUS

Batch size scaling: When reducing NUM_GPUS, reduce batch_size proportionally to avoid OOM:

On RTX 4090 (24GB): reduce batch size to 4-8 and consider enabling gradient checkpointing.

Evaluation Only

To evaluate without further training, set VAL_ONLY=True:

SFT_MODEL_PATH="./checkpoints/post-rl_spatial"
VAL_ONLY=True
DATASET_NAME="libero_spatial"

bash scripts/run_libero_rl_training.sh \
  --sft-model-path $SFT_MODEL_PATH \
  --val-only $VAL_ONLY \
  --dataset-name $DATASET_NAME

The script evaluates all 10 tasks in the selected suite and reports per-task and average success rates.

Results

Simulation (LIBERO Benchmark)

Notable: LaST-R1 beats πRL (99.8% vs 98.3%) while πRL uses full trajectories for warmup versus LaST-R1's 1 trajectory per task.

Real-World (Franka Research 3)

Four real manipulation tasks, 20 evaluations each:

These numbers show LAPO RL actually learns new skills through exploration — not just "refining" the warmup policy.

Out-of-Distribution Generalization

When swapping objects (unseen), backgrounds, and lighting, LaST-R1 drops by just 8% on average — while the Action-Only PPO baseline drops over 20%. Latent reasoning helps the model understand physics rather than memorizing visual patterns.

For related RL techniques, see: SimpleVLA-RL — Scaling RL for VLA Models.

Comparison with Other Approaches

Why does Latent CoT matter so much? Sample efficiency. When the model has internal reasoning structure, gradients from RL have clear direction instead of random exploration across the action space. Result: faster convergence with fewer rollouts.

For more on action-level Chain-of-Thought in VLA, see: Action CoT for VLA with LeRobot.

For how FORCE approaches similar problems with value-calibrated warmup: FORCE: Fine-tune VLA with RL and Value-Calibrated Warm-Up.

Practical Pitfalls

1. Reward design is the real bottleneck

LaST-R1 uses binary reward (0/1) for LIBERO — easy to implement since the simulator has ground truth. For real robots, you need a vision-based task completion detector or dense reward engineering. This is where most time is spent in practice.

For reward design strategies for manipulation: ProcVLM: Dense Rewards for VLA RL.

2. RL training requires substantial GPU resources

Online RL with LAPO needs parallel rollout collection alongside policy updates — this is why veRL is used (distributed rollout support). Single-GPU training is dramatically slower than 8-GPU setup. Rough estimate: training to convergence on 1 suite (~10 tasks) takes 12-24 hours on 8× A100.

3. DINOv3 top-k is an important hyperparameter

k=2560 is selected to match the VLA embedding size with Qwen3-VL-4B. If you switch backbone (e.g., Qwen3-VL-7B), you need to recalibrate k for the new embedding dimension. Wrong k → latent representation truncated or padded incorrectly.

4. Adaptive CoT needs calibration for new domains

Confidence threshold p≥0.99 works well for LIBERO but may need adjustment for new domains. Too high → model always uses maximum reasoning length (wasted compute). Too low → model terminates reasoning before gathering enough information.

5. Use LoRA for real-world deployment

The paper mentions using LoRA fine-tuning for real-world deployment to reduce VRAM. If deploying on Jetson Orin or standard workstations, consider LoRA rank 16-32 instead of full fine-tuning.

When to Use LaST-R1

Use LaST-R1 when:

• You have few demonstrations (< 5 demos/task) but need high accuracy
• Task requires contact-rich manipulation (inserting, screwing, pulling zippers)
• Policy needs to generalize to unseen objects and backgrounds
• You have compute budget for online RL (≥ 2× A100/H100)

You may not need LaST-R1 when:

• You have many demos and the task is simple → standard SFT is sufficient
• Compute budget is limited → SimpleVLA-RL or GRAPE are lighter
• Task is fully deterministic and repetitive → classical controller is more efficient

Summary

LaST-R1 addresses a fundamental weakness of VLA + RL: instead of only optimizing actions, LAPO jointly optimizes both the internal physical reasoning process and the resulting actions. Three core innovations:

1. DINOv3 latent targets — physically-grounded representation stronger than Q-Former or global pooling
2. LAPO — joint optimization of latent reasoning and action with Gaussian approximation for continuous latent likelihood
3. Adaptive CoT — automatic reasoning length adjustment per task, no wasted compute

The 99.8% result on LIBERO with 1 demo per task is among the most compelling in manipulation VLA as of mid-2026. Particularly important is the generalization improvement — the model learns to understand physics rather than memorize visual patterns — a prerequisite for real-world deployment where conditions are always changing.

Related Posts

• SimpleVLA-RL: Scaling RL for VLA from LIBERO to Real Robots
• Action CoT for VLA: Chain-of-Thought in Action Generation with LeRobot
• FORCE: Fine-tune VLA with RL, Value-Calibrated Warm-Up and Self-Distillation