RL²-VLA: Offline RL Latent Steering Boosts VLA Manipulation +26% at Test Time, No Retraining

Vision-Language-Action (VLA) models like π0 and OpenVLA excel within their training distribution, but performance degrades sharply on out-of-distribution (OOD) tasks — different instruction phrasings, novel object positions, subtle environment changes. The conventional fix is RL fine-tuning, but that means touching billions of parameters, running many online rollouts, and risking catastrophic forgetting.

RL²-VLA from MARMot Lab (NUS) takes a different route: keep the base VLA frozen, and attach a lightweight offline RL steering module that operates in the VLA's latent space — intervening only when the base model is predicted to fail. The result: +26.7% success rate on a real PiperX robot arm, +14.7% on SIMPLER benchmark, zero changes to the base VLA weights.

• Paper: RL²-VLA (arXiv 2607.26991)
• Code: github.com/marmotlab/RL2-VLA
• Project page: rl2-vla.github.io
• Models: huggingface.co/rl2-vla

Why VLAs Fail Out-of-Distribution

VLAs learn via behavior cloning — maximizing likelihood over demonstrations. This pushes the model toward the dominant action mode in training data. When deployment introduces slight variations (synonym instructions, novel object placement, camera angle differences), the model gets anchored to learned modes that may no longer be optimal.

RL is the natural fix: it can discover high-value behaviors beyond demonstration modes. But full RL fine-tuning of a billion-parameter VLA is expensive: it requires many online rollouts, careful reward engineering, and can overwrite the model's broad pretrained knowledge.

RL²-VLA decouples the two concerns: the VLA retains broad semantic knowledge, while a small, targeted RL module handles local action diversity when needed.

Architecture: Three Coordinated Modules

RL²-VLA has three independent components that coordinate at inference time:

1. SAFE — Adaptive Failure Detector

SAFE (Scalable Adaptive Failure Estimator) is a small LSTM conditioned on internal latent features of the VLA action expert — not raw image observations. It learns to distinguish trajectories heading toward success from those heading toward failure.

Key innovation: SAFE uses conformal prediction to calibrate detection thresholds. Rather than a fixed threshold, conformal prediction automatically sets per-task bands such that successful rollouts stay below the threshold with probability 1-α. This prevents unnecessary steering when the VLA is already performing correctly.

Calibration process:

1. Collect rollouts from the base VLA (60% train / 40% validation split)
2. Train SAFE to predict a failure score ∈ [0, 1] at each timestep
3. Use successful validation rollouts to fit conformal prediction bands
4. Per-task thresholds ensure the false-positive steering rate stays bounded

2. QAM — Offline RL Steering Policy

QAM (Q-learning with Adjoint Matching) is the core of the approach. It is a lightweight flow-matching policy trained with offline RL, conditioned on latent representations from the VLA action expert.

The technical challenge: Why "Adjoint Matching"? Flow-matching policies generate actions through multi-step ODE integration. Backpropagating through those steps is numerically unstable — gradients explode or vanish. Adjoint Matching computes gradients via a reverse ODE, which is stable and avoids backpropagating through the full forward pass.

Composition at inference: When SAFE detects a predicted failure, the action expert's flow velocity and QAM's steering velocity are composed:

v̂ = w · v_VLA + (1 - w) · v_RL

The weight w is sampled from Gaussian(μ=0.5, σ=0.25), so the VLA remains the dominant contributor but receives RL-guided nudges toward alternative behaviors.

Why latent space? Rather than intervening in observation space or raw action space, QAM operates in the VLA's expressive internal latents. These features already encode rich context (scene semantics, instruction meaning, robot state) — the RL policy can condition on this to produce contextually appropriate guidance.

3. CoVer — Action Verifier

CoVer (Contrastive Verifier) is an external reward model trained with contrastive learning to score candidate actions. When multiple action samples are generated (multi-sample test-time scaling), CoVer ranks them and selects the best for execution.

CoVer does not generate actions and requires no retraining when switching VLA backbones. It simply asks: given the current state, which candidate action is most likely to succeed?

Test-Time Scaling Laws

One of the paper's key findings is that success and failure states follow different scaling laws:

• During success: adding diversity/more samples does not help — it may perturb already-accurate actions.
• During failure: adding diversity significantly helps — exploring alternative modes finds a path to success.

This asymmetry is why adaptive steering (trigger only on predicted failure) consistently outperforms always-on steering. SAFE identifies the moments that actually benefit from intervention.

Scaling experiment results: using 8 instruction rephrases × 5 samples per rephrase achieves +18.6% improvement — but only with adaptive triggering. Without it, many samples waste compute or degrade already-good actions.

Installation and Setup

System Requirements

• Linux Ubuntu 20.04 or 22.04
• Python 3.10
• CUDA-capable GPU (H100, A6000, or RTX 5090 recommended; RTX 3090+ sufficient for evaluation)
• VRAM: ≥24GB to load the π0 base model

Clone the Repository

git clone --recurse-submodules https://github.com/marmotlab/RL2-VLA.git
cd RL2-VLA

The --recurse-submodules flag is required — the SAFE component lives in a submodule.

Create the Conda Environment

conda create -n rl2 python=3.10
conda activate rl2

Install Dependencies

# Install all dependencies for SIMPLER simulation + π0 integration
bash RL2_CoVer_VLA/env_simpler_pi.sh

This script installs lerobot, simpler-env, pytorch, diffusers, and related packages. Expect 10–20 minutes depending on network speed.

Download Pretrained Models

# QAM steering policy
huggingface-cli download rl2-vla/qam-steering-policy --local-dir ./checkpoints/qam

# CoVer action verifier
huggingface-cli download cover-vla/cover-vla-bridge --local-dir ./checkpoints/cover

The SAFE detector is included in the submodule and does not need a separate download.

Training Pipeline (Custom Data)

RL²-VLA supports training on your own dataset. The pipeline has four steps:

Step 1: Extract VLA Latents

# Extract internal representations from the frozen VLA action expert
python RL2_CoVer_VLA/extract_latents.py \
    --dataset bridge_v2 \
    --vla_checkpoint path/to/pi0 \
    --output_dir data/latents/

Latents are extracted from the hidden states of the action expert transformer — not the visual encoder or language encoder. These high-level features compress task context efficiently.

Step 2: Train the QAM Steering Policy

python RL2_CoVer_VLA/train_qam.py \
    --latent_dir data/latents/ \
    --batch_size 256 \
    --lr 3e-4 \
    --epochs 100 \
    --output_dir checkpoints/qam/

Training uses Q-learning with Adjoint Matching:

• Q-values estimated from offline data (no online rollouts needed)
• Policy updates via reverse ODE for numerically stable gradient computation
• Conditioned on latents so the policy is context-aware

On an A100, training on BridgeV2 takes approximately 2–4 hours.

Step 3: Train the SAFE Failure Detector

# Collect rollouts from base VLA
python RL2_CoVer_VLA/collect_rollouts.py \
    --env simpler \
    --tasks bridge_tasks \
    --n_rollouts 2000 \
    --output_dir data/rollouts/

# Train SAFE LSTM
python RL2_CoVer_VLA/train_safe.py \
    --rollout_dir data/rollouts/ \
    --train_ratio 0.6 \
    --output_dir checkpoints/safe/

Step 4: Calibrate Conformal Prediction

python RL2_CoVer_VLA/calibrate_safe.py \
    --rollout_dir data/rollouts/ \
    --safe_checkpoint checkpoints/safe/ \
    --alpha 0.1 \
    --output_dir checkpoints/safe/thresholds/

alpha=0.1 means that with 90% probability, a successful rollout will not trigger a false-positive steering activation.

Inference and Evaluation

SIMPLER Benchmark Evaluation

conda activate rl2

# Adaptive steering (recommended) — steer only when SAFE predicts failure
bash RL2_CoVer_VLA/simpler/bashes/eval_rl2_compose_adaptive.sh

# Always-on steering — steer every timestep (comparison baseline)
bash RL2_CoVer_VLA/simpler/bashes/eval_rl2_compose_always.sh

# Rephrase-only baseline — no RL steering
bash RL2_CoVer_VLA/simpler/bashes/eval_rephrase.sh

Summarize Results

python RL2_CoVer_VLA/simpler/bashes/summarize_logs.py \
    --log_dir logs/simpler_eval/

Output is a per-task success rate table plus overall averages.

Integration with a Custom Robot

import torch
from rl2_vla import RL2Pipeline

# Load the full pipeline
pipeline = RL2Pipeline.from_pretrained(
    vla_checkpoint="path/to/pi0",
    qam_checkpoint="checkpoints/qam/",
    safe_checkpoint="checkpoints/safe/",
    cover_checkpoint="checkpoints/cover/",
    adaptive=True  # only steer when needed
)

# Inference loop
obs = robot.get_observation()
instruction = "pick up the red cup and place it in the bowl"

action = pipeline.predict(
    observation=obs,
    instruction=instruction,
    n_samples=5  # candidate actions for CoVer to rank
)

robot.execute(action)

Benchmark Results

SIMPLER Simulation

SIMPLER is a physics-based simulation benchmark with tasks including pick-and-place, push, and rotate. RL²-VLA was evaluated on OOD instruction prompts — same underlying tasks but described differently from training data.

Adaptive steering consistently outperforms always-on, because it avoids perturbing actions the VLA already has correct.

PolaRiS Benchmark

PolaRiS tests π0.5 (a stronger base model than π0) on harder OOD scenarios:

Real Robot: PiperX Arm

The most important results — hardware validation. PiperX is a 6-DOF desktop manipulator with a wrist camera:

Real-robot improvements exceed simulation improvements because OOD gaps are larger in the physical world. The adaptive trigger correctly identifies where steering matters most.

Comparison with Related Approaches

Several directions attempt to improve VLA performance at test time. RL²-VLA's positioning:

Compared to ResVLA, which also uses a residual approach but focuses on intent anchoring rather than failure detection — RL²-VLA's adaptive triggering is a key differentiator.

Compared to direct test-time compute scaling — RL²-VLA is a specialized form of test-time compute: instead of naive sample scaling, it applies RL-guided steering to improve the quality of each sample, then uses CoVer to select the best.

For a deeper foundation on VLA architecture, see the OpenVLA deep dive.

Ablation Analysis

The paper rigorously ablates each component:

Latent conditioning vs. raw observation: Using VLA latents outperforms raw observations by +4.2% — latents compress richer task context.

RL vs. behavior cloning for steering: QAM (RL-trained) outperforms a BC-trained equivalent by +4.5% — RL discovers action diversity beyond demonstration modes.

Adaptive vs. always-on: Adaptive outperforms always-on by +2.3% on OOD tasks, while also avoiding degradation on in-distribution tasks (always-on can harm already-correct actions).

CoVer verification: Adding CoVer selection during multi-sampling adds +1.8% on top of steering alone.

Limitations and Future Directions

Current limitations:

• QAM is trained on BridgeV2/DROID — performance may vary on domains far from these datasets
• SAFE requires calibration rollouts — a new robot type needs additional data collection
• High VRAM requirement (≥24GB) due to the π0 base model

Interesting extensions:

• Apply to other VLA backbones: OpenVLA-OFT, RoboVLMs
• Extend to more complex manipulation: bimanual, contact-rich tasks
• Online adaptation: incrementally update QAM from real robot rollouts

Conclusion

RL²-VLA demonstrates that retraining the VLA is not necessary for significant performance improvements. By training a small latent-space steering module with offline RL and triggering it intelligently (only when failure is predicted), the framework achieves +26.7% success rate on a real robot arm — a substantial gain at low compute cost.

The three-component architecture — SAFE + QAM + CoVer — cleanly separates concerns: SAFE knows when to intervene, QAM knows how to steer, CoVer knows which action to select. This modularity allows independent upgrades as stronger base VLAs emerge.

With test-time compute scaling becoming a major trend in robotics, RL²-VLA offers a principled approach: not naive sample scaling, but intelligent RL-guided latent steering with adaptive triggering and verified selection.

Related Posts

• ResVLA: Residual Diffusion Bridge for Anchoring VLA Policy from Intent
• Robo-ValueRL: Deploy VLA-RL Precision Manipulation for Humanoid Robots
• Expo-FT Pi0.5: Online RL Fine-tune VLA in 19 Minutes