ResVLA (ICML 2026): From Noise to Intent — Anchoring VLA Policies with Residual Diffusion Bridges

Picture teaching a robot to pick up a cup. You say: "Grab the red cup on the left side of the table." Your brain processes this in milliseconds, then your hand executes — but not by computing every micromovement from random noise. Your brain first locks onto the intent (pick up the red cup, bring it up), and your muscles only need to refine the physical details.

This is exactly the problem ResVLA (ICML 2026) solves — and its approach is elegant enough to reshape how we design diffusion-based VLA policies.

Paper: From Noise to Intent: Anchoring Generative VLA Policies with Residual Bridges
GitHub: 4DVLab/ResVLA
Authors: Yiming Zhong, Yaoyu He, Zemin Yang, Pengfei Tian, Yifan Huang, Qingqiu Huang, Xinge Zhu, Yuexin Ma (ShanghaiTech University, Morphic Robotics, CUHK)

Figure 1: ResVLA shifts from "Generation-from-Noise" to "Refinement-from-Intent". Instead of starting from pure Gaussian noise, the model first predicts global intent, then refines the residual — source: arXiv 2604.21391

1. The Problem: What "Generation-from-Noise" Wastes

Every diffusion-based VLA today — from Diffusion Policy to π₀ — operates under the "Generation-from-Noise" paradigm: initialize from pure Gaussian noise N(0, I), then iteratively denoise to produce an action sequence.

This creates two serious problems:

Problem 1: Representation Inefficiency

Gaussian noise carries zero information about the task. This means the model must spend much of its capacity "rediscovering" the global intent — that this is a grasping task, not a pushing one; that the target is the red cup, not the blue one; that the trajectory should go left across the table.

This is redundant work. The VLM backbone already understands the language instruction and visual scene. Why does the diffusion head need to re-invent all of that from pure noise?

Problem 2: Loss Collapse

When the source distribution N(0, I) is completely independent of the conditioning input c (language + image), gradient flow from the loss back to the condition encoder becomes nearly zero. ResVLA proves this formally in Theorem 3.3: mutual information I(x₀; c) = 0 when the source is pure noise → gradients short-circuit and cannot reach the language condition encoder.

The practical consequence: the model learns generic average actions but doesn't follow specific language instructions. If you've ever trained a VLA and found the robot performing the same motion regardless of whether you say "pick up" or "put down" — that's Loss Collapse.

2. The Core Insight: Motion = Intent + Dynamics

ResVLA builds on a deceptively simple but profound observation:

Robot motion naturally decomposes into two components: global intent and local dynamics.

• Global intent (low-frequency, deterministic): the overall trajectory — the robot moves toward the cup, opens the gripper, descends, closes. This is the "what" and "where" of the task. It varies slowly and can be predicted from language + vision.
• Local dynamics (high-frequency, stochastic): the microadjustments — exact force magnitudes, wrist orientation tweaks, millisecond-level corrections during contact. This is the "how exactly" at the physical level. It varies fast and depends on immediate environmental state.

This decomposition maps directly onto the separation between trajectory planning (high-level, semantically driven) and reactive control (low-level, physics-dependent) — a distinction classical robotics has always maintained. ResVLA brings this principle into a unified learning framework.

So instead of learning both from noise, ResVLA proposes:

1. Predict intent deterministically from VLM features (Stage 1)
2. Refine the residual — the remaining stochastic component — from that anchored intent (Stage 2)

3. Spectral Analysis: DCT Separates High and Low Frequencies

To implement this idea, ResVLA uses the Discrete Cosine Transform (DCT) — the same tool JPEG uses to compress images (separating "overall shape" from "fine noise detail").

Given action sequence x ∈ ℝ^(T×D) (T timesteps, D action dimensions):

x_S = DCT_S(x)    # Low-frequency components: Semantic subspace (global intent)
x_E = DCT_E(x)    # High-frequency components: Execution subspace (local dynamics)

# The two subspaces are complementary and sum to the full action:
x = x_S + x_E

Why DCT over wavelets or learned PCA? Three practical reasons:

1. Fixed basis, no learning: DCT uses orthogonal cosine functions as basis vectors — zero extra parameters to optimize.
2. Strong energy compaction: >80% of a smooth trajectory's variance concentrates in the first ~30% of low-freq modes.
3. Computationally trivial: FFT-based DCT is O(N log N), negligible compared to the VLM forward pass.

4. ResVLA Architecture: Two Cascading Stages

ResVLA is built on Qwen3-VL-2B-Instruct as the VLM backbone (2B parameters — significantly lighter than π₀'s 3B or GR00T N1's 7B+), with two key modules:

Stage 1: Intent Anchoring Module

This module takes VLM features from Qwen3-VL and deterministically predicts the low-frequency action component:

μ_prior(c) = IntentAnchorModule(VLM_features(image, language_instruction))

ResVLA then constructs a condition-dependent source distribution (replacing pure N(0, I)):

p₀(x|c) = 𝒩(x; μ_prior(c), σ²_min · I)

This is the key architectural decision: instead of starting diffusion from pure noise, we start from a tight Gaussian centered on the predicted intent — very close to the target action distribution. σ_min is small (e.g., 0.001), so the variance around the anchor is minimal.

Stage 2: Residual Diffusion Bridge

From the anchored distribution, the Residual Diffusion Bridge learns the transport path from the anchored source → target action distribution, using Conditional Flow Matching (CFM):

# Velocity field with optimal transport path:
v_t(x_t) = x₁ - x₀
# x₁ = ground truth action
# x₀ ~ p₀(x|c) = 𝒩(μ_prior(c), σ²_min · I)

The velocity field now only needs to learn a small residual Δx = x_gt - x₀. Because x₀ is already close to x₁ (thanks to the anchor), the transport cost (kinetic energy) is minimized → straighter paths → fewer integration steps → NFE=1 is viable.

This is why the module is called a "Residual Diffusion Bridge" — it bridges the short gap from intent to action, rather than traversing the full distance from noise.

5. Loss Function: Bi-Level Optimization

The training objective combines two cooperative terms:

# L_total = λ_sem × L_semantic + L_residual

# Term 1: Intent Anchoring Loss
# Aligns intent prediction with the low-frequency ground truth
L_semantic = ||μ_prior(c) - x_S||²   # MSE on semantic subspace

# Term 2: Residual Flow Matching Loss
# Learns velocity field to transport from anchor to target
L_residual = E_{t, x_t} ||v_θ(x_t, t, c) - (x_gt - x₀)||²

The two losses work cooperatively:

• Better L_semantic → more accurate anchor → smaller gap between x₀ and x₁ → easier L_residual
• L_residual converges → model better understands the execution space → feeds back into improved intent quality

Unlike two-stage pipelines (pretrain intent, then fine-tune residual), ResVLA trains end-to-end — both losses share the VLM backbone and influence each other through shared gradients.

6. Inference: "Predict-Refine" Flow

The inference procedure is highly intuitive, in two steps:

# Step 1: Predict intent (fully deterministic, no stochasticity)
mu_prior = intent_anchoring_module(vl_features(image, language))

# Step 2: Sample from the anchored distribution
x_0 = mu_prior + sigma_min * torch.randn_like(mu_prior)

# Step 3: Refine residual via numerical ODE integration
# Euler method (NFE=1) — sufficient because the residual is small:
x_hat = x_0 + v_theta(x_0, t=0.0, condition=c)

# Or multi-step for higher accuracy:
# x_hat = ode_solve(v_theta, x_0, t_span=[0, 1], method='euler', n_steps=4)

Why NFE=1 works: because the transport path from anchor to target is short and nearly straight (the anchor is already close to the target), a single Euler step achieves >70% success rate — equivalent to what baselines need NFE=4 to reach. This matters enormously for real-time robot control, where inference latency must stay under 50ms.

7. Environment Setup

System Requirements

Python 3.10
PyTorch 2.6.0 + CUDA 12.4
FlashAttention (required for Qwen3-VL backend)
Training GPU: at least 1× A100 40GB
Eval GPU: 1× RTX 3090 24GB or equivalent

Installation

git clone https://github.com/4DVLab/ResVLA.git
cd ResVLA

# Create conda environment
conda create -n resvla python=3.10 -y
conda activate resvla

# Install PyTorch with CUDA 12.4
pip install torch==2.6.0 torchvision==0.21.0 \
  --index-url https://download.pytorch.org/whl/cu124

# Install project dependencies
pip install -r requirements.txt
pip install -e .

# Install FlashAttention (compiles from source — takes 10-15 min)
pip install flash-attn --no-build-isolation

Download VLM Backbone

ResVLA uses Qwen3-VL-2B-Instruct as its vision-language backbone:

python -c "
from huggingface_hub import snapshot_download
snapshot_download(
    'Qwen/Qwen3-VL-2B-Instruct',
    local_dir='./checkpoints/qwen3vl'
)
"

8. Data Preparation

LIBERO (Standard Manipulation Benchmark)

LIBERO contains 90 diverse tasks (LIBERO-90) with ~50K expert demonstrations — the standard benchmark for tabletop manipulation research.

# Automatically downloads and prepares LIBERO datasets
bash examples/LIBERO/data_preparation.sh

This script downloads raw LIBERO data and converts it to LeRobot format — the same format used across the LeRobot ecosystem.

SimplerEnv / Open X-Embodiment

For training and evaluation on SimplerEnv (Google Robot and WidowX):

# Required directory structure:
playground/
└── Datasets/
    └── SimplerEnv/
        ├── bridge/    # Bridge dataset in LeRobot format
        └── fractal/   # Fractal20 dataset in LeRobot format

# Download bridge dataset:
python -c "
from huggingface_hub import snapshot_download
snapshot_download('lerobot/bridge_v2',
    local_dir='playground/Datasets/SimplerEnv/bridge')
"

# Download fractal dataset:
python -c "
from huggingface_hub import snapshot_download
snapshot_download('lerobot/fractal20220817_data',
    local_dir='playground/Datasets/SimplerEnv/fractal')
"

9. Training

Training on LIBERO

# Run the default training script (adjust GPU count and batch size for your hardware)
bash examples/LIBERO/train_files/run_libero_train_resVLA.sh

Key parameters inside the script:

CUDA_VISIBLE_DEVICES=0,1,2,3 \
torchrun --nproc_per_node=4 \
  train_resVLA.py \
  --model_name_or_path ./checkpoints/qwen3vl \
  --dataset_path playground/Datasets/LIBERO/libero_90 \
  --batch_size 16 \
  --learning_rate 1e-4 \
  --num_epochs 20 \
  --lambda_sem 1.0 \          # Weight for L_semantic
  --dct_low_freq_ratio 0.3 \  # First 30% of DCT modes = semantic subspace
  --sigma_min 0.001 \         # Anchored distribution variance
  --output_dir ./outputs/libero

Parameters to understand deeply:

• --dct_low_freq_ratio 0.3: the fraction of low-frequency DCT modes treated as the semantic subspace. The paper tested multiple values; 0.3 works best on LIBERO because robot trajectories tend to be smooth.
• --lambda_sem 1.0: balances L_semantic and L_residual. Increase → tighter anchor but harder residual task; decrease → looser anchor but more expressive residual.
• --sigma_min 0.001: variance of the anchored distribution. Smaller → tighter anchor, but if intent prediction is wrong there's less "safety net" for the residual.

Training on SimplerEnv / OXE

bash examples/SimplerEnv/train_files/run_oxe_train_ResVLA.sh

Estimated training time (4× A100 40GB):

• LIBERO-90: ~12–15 hours
• SimplerEnv/OXE: ~24–36 hours

10. Using Pretrained Checkpoints

For quick evaluation without training from scratch:

# LIBERO checkpoint (Qwen3-VL-2B, trained on LIBERO-90)
python -c "
from huggingface_hub import snapshot_download
snapshot_download(
    'GaussionZhong/resvla_libero_all_2B',
    local_dir='./checkpoints/libero'
)
"

# SimplerEnv checkpoint (trained on Bridge + Fractal)
python -c "
from huggingface_hub import snapshot_download
snapshot_download(
    'GaussionZhong/resvla_simpler_env_2B',
    local_dir='./checkpoints/simpler'
)
"

11. Evaluation and Inference

Evaluating on LIBERO

ResVLA uses a client-server architecture: the policy server runs the model, and the simulator client calls the API to get actions.

# Terminal 1: Start the ResVLA policy server
python resVLA/policy_server.py \
  --checkpoint ./checkpoints/libero \
  --host 0.0.0.0 \
  --port 8000

# Terminal 2: Run LIBERO simulator and query the policy
cd examples/LIBERO
python eval_libero.py \
  --server_url http://localhost:8000 \
  --num_episodes 50

Full details: see examples/LIBERO/README.md.

LIBERO-Plus (Robustness Evaluation)

LIBERO-Plus extends LIBERO with three types of perturbation to test real-world robustness:

1. Language variation: rephrase instructions with synonyms and restructured sentences
2. Robot embodiment: swap to a different robot arm — tests cross-embodiment transfer
3. Layout changes: shift object positions relative to the demonstration

# Run with specific perturbation type:
python eval_libero_plus.py \
  --server_url http://localhost:8000 \
  --perturbation_type language   # language | embodiment | layout

12. Benchmark Results

LIBERO (Standard Benchmark — Table 2)

ResVLA reaches 96.3% — matching OpenVLA-OFT nearly perfectly, but without pretraining on large-scale internet data. ResVLA trains from scratch on LIBERO-90 alone, while several baselines rely on Internet-scale pretrained weights.

LIBERO-Plus (Robustness — The Numbers That Matter)

The embodiment perturbation number is striking: π₀ collapses to 6.0% when switching robot arms, while ResVLA holds at 59.9% — nearly 10× better. The reason: intent anchoring makes the model understand the task at the semantic level, decoupled from the specific kinematics of any one robot.

SimplerEnv (Cross-Embodiment, Real Dataset)

Real-World ALOHA (Bimanual Robot)

Multi-stage task: pick → handover → place (3 consecutive stages, all must succeed)

13. Expert Perspective: Why ResVLA Matters Beyond the Numbers

A senior robotics engineer reading ResVLA will immediately notice three things a beginner might miss:

1. This isn't "faster diffusion" — it's a change in inductive bias

NFE=1 inference is a side effect, not the primary goal. The core contribution is that ResVLA places the model in the right region of action space before refinement begins. This single change affects everything — generalization, convergence, robustness — not just speed.

2. Loss Collapse explains why VLAs ignore language

If you've trained a VLA and seen it ignore instructions, you might have blamed data quality or learning rate. ResVLA identifies a deeper architectural cause: when the noise source is independent of the condition, gradients physically cannot reach the language encoder. You cannot fix this with data alone — you need to change the source distribution.

3. DCT is a deliberately un-fancy choice

Beginners often assume you need a learned decomposition (auto-encoder, online PCA) to cleanly separate intent from dynamics. DCT proves that a fixed basis works well because robot trajectories are inherently smooth and low-frequency energy dominant. Simple tools often work when the problem structure is right.

For context on how VLA policies fit into the broader manipulation ecosystem, see the VLA manipulation series.

14. Conclusion

ResVLA provides a clean answer to: "Why force diffusion to learn from pure noise when the VLM already understands the task?"

The Residual Diffusion Bridge doesn't just push benchmark numbers — it improves robustness (more important than accuracy for real deployment), convergence speed (less data needed to reach the same performance), and inference efficiency (NFE=1 is practical, not just a theoretical curiosity).

The Refinement-from-Intent paradigm has obvious extension potential beyond tabletop manipulation. The open question: can the same approach work for whole-body humanoid control — where the gap between semantic intent (climb stairs) and local dynamics (each joint microadjustment) is even larger than in table-top tasks?

Related Posts

• Diffusion Policy: The Foundation of Modern Robot Manipulation
• VLA Models: From Language to Robot Action
• LeRobot VLA Framework: Complete Guide
• Whole-Body VLA and Latest WBC 2026