VLAC Guide: Train VLA with a Critic to Reach 90% in 200 Episodes

One of the oldest unsolved problems in robot learning is deceptively simple: how does a robot know if it's making progress? The standard answer is hand-crafted reward functions — mathematical formulas measuring distance, contact force, object position. But every new task demands a new reward function, consuming weeks of engineering time for each deployment.

VLAC (Vision-Language-Action-Critic) — published by Shanghai AI Lab in September 2025 — attacks this problem from a fundamentally different angle: instead of engineering rewards, teach a model to assess progress from data. The result? Success rate climbs from ~30% to ~90% within 200 real-world episodes, with zero reward engineering per task.

The Root Problem: Why Is Real-World RL Hard?

Reinforcement learning for physical robots faces two interlocking bottlenecks:

1. Sparse reward: The robot only knows success or failure at episode end. Over 200 episodes, that might mean 190 complete failures before learning anything useful — like teaching a child to play basketball by only saying "wrong" or "right" at the final buzzer.

2. Reward engineering is expensive: For RL to converge fast, you need dense reward — a continuous signal at every timestep. Writing accurate reward functions for complex manipulation (folding a napkin, scooping rice into a bowl) requires deep domain knowledge and lengthy debugging.

VLAC attacks both simultaneously.

What Is VLAC?

VLAC is a process reward model (PRM) built on InternVL — a powerful vision-language model from Shanghai AI Lab. Instead of a hand-written reward function, VLAC learns to ask: "Between image A and image B, did the robot get closer to the goal or farther?"

The key innovation is pair-wise progress understanding: VLAC never evaluates a single frame in isolation. It compares consecutive image pairs within a trajectory and outputs:

• Progress delta (Δp): A signed scalar — positive means forward progress, negative means regression
• Done signal: Binary flag — has the task completed?
• Action tokens: VLAC can also generate actions directly (acting as a pure VLA policy)

All three outputs come from a single model, controlled by prompt switching between critic and actor roles.

Architecture Deep Dive

Input: [Frame t-1] + [Frame t] + [Language goal] → InternVL backbone
                                                           ↓
                                                 Autoregressive decoder
                                               ↙          ↓          ↘
                                       Progress Δ    Done signal    Action tokens
                                       (reward)      (termination)  (delta EE pose)

Foundation: InternVL

InternVL was chosen for its strong visual understanding and native multi-image input support — essential for comparing image pairs. It is fine-tuned on robotics data so the model can "speak" the language of physical manipulation.

Pair-wise comparison as reward signal

Instead of asking "where is the robot?", VLAC asks "how did the robot change from t-1 to t?" Framing the problem around change rather than absolute state makes the model robust to background variation, lighting shifts, and camera angle differences across setups.

Structured output format

Action tokens follow a delta end-effector pose format (Δx, Δy, Δz, ΔRx, ΔRy, ΔRz, gripper) — generic enough to transfer across different robot hardware.

Training Data

One of VLAC's strongest assets is diverse, large-scale training data:

How progress delta labels are generated — without human annotation

This is the clever part: labels are derived from temporal ordering automatically:

• Frame 5 vs Frame 1: progress ≈ +0.4 (large forward progress)
• Frame 3 vs Frame 5: progress ≈ -0.2 (slight regression)
• Last frame vs second-to-last: progress ≈ +0.9 on successful episodes

Negative samples round out the training: unrelated image pairs → progress ≈ 0, mismatched task description → model learns to reject irrelevant inputs.

The Real-World RL Loop

VLAC is not a standalone reward model — it integrates into an asynchronous real-world RL loop:

┌──────────────────────────────────────────────┐
│         Asynchronous RL Loop                 │
│                                              │
│  Robot executes → collect (obs, action)      │
│         ↓                                    │
│  VLAC Critic evaluates progress              │
│  → outputs dense reward Δp                   │
│         ↓                                    │
│  Policy update (offline or online RL)        │
│         ↓                                    │
│  Updated policy deployed to robot            │
│         ↑                                    │
│  [Human-in-the-loop can intervene here]      │
└──────────────────────────────────────────────┘

Asynchronous means the robot never stops to wait for training — it keeps collecting data while the policy updates on a separate machine.

Human-in-the-Loop: Three-Tier Protocol

The most distinctive aspect of VLAC is its structured three-tier human intervention protocol, applied progressively as training matures:

Tier 1: Offline Demonstration Replay

When: Early in training when the policy knows nothing.

How: Pre-populate the replay buffer with human demonstrations. The policy does not imitate demos directly — instead, VLAC critic re-annotates each demo with progress rewards, and the policy learns from (obs, action, reward) tuples as standard RL data.

Why it works: Human demos are complete, coherent trajectories. The robot learns what progress looks like from the very first episode, instead of random exploration from scratch.

Tier 2: Return and Explore

When: Policy has learned the rough shape of the task but is stuck in specific failure modes.

How: An operator monitors in real-time. When the robot enters a failure state it cannot escape (e.g., object knocked out of reach), the operator manually resets the robot to that exact failure state and lets it retry multiple times.

Why it works: Targeted exploration coverage. The robot doesn't need to stumble upon rare failure states by chance — the operator teleports it there on demand.

Tier 3: Human Guided Explore

When: Sub-behaviors exist that the policy cannot discover through exploration alone (e.g., approaching an object from a difficult angle).

How: The operator teleoperate the robot through the difficult segment, then hands back control to the policy. This data enters the replay buffer with VLAC-computed rewards.

Why it works: Provides scaffolding for complex motor skills. Pure exploration might take thousands of episodes to discover them organically.

Installation and Usage

System Requirements

• Python ≥ 3.9
• PyTorch ≥ 2.0
• CUDA 12 (recommended)
• GPU VRAM: 8 GB+ for VLAC-2B, 24 GB+ for VLAC-8B

Installation

git clone https://github.com/InternRobotics/VLAC.git
cd VLAC
pip install -e .

Download Models

Two variants are available on HuggingFace:

from huggingface_hub import snapshot_download

# VLAC-2B — lightweight, suitable for fast inference
snapshot_download("InternRobotics/VLAC", local_dir="./models/vlac-2b")

# VLAC-8B — higher accuracy, needs powerful GPU
snapshot_download("InternRobotics/VLAC-8b", local_dir="./models/vlac-8b")

Using VLAC as a Critic (trajectory evaluation)

from vlac import VLACCritic

# Initialize critic
critic = VLACCritic.from_pretrained("InternRobotics/VLAC")

# Feed in an image sequence and task description
result = critic.web_trajectory_critic(
    images=["frame_001.jpg", "frame_002.jpg", "frame_003.jpg"],
    task_description="Pick up the bowl and place it on the tray",
)

# Results
print(result["progress_deltas"])  # [0.1, 0.3, 0.5, ...]
print(result["done_signals"])     # [False, False, True, ...]
print(result["value_scores"])     # Overall trajectory quality score

Using VLAC as a Pure VLA Policy (action generation)

from vlac import VLACPolicy

policy = VLACPolicy.from_pretrained("InternRobotics/VLAC")

# Predict action from current observation
obs = {
    "image": current_frame,
    "task": "Unfold the mat on the table"
}
action = policy.predict(obs)
# action = [Δx, Δy, Δz, ΔRx, ΔRy, ΔRz, gripper_open]

Integrating VLAC into an RL Loop

from vlac import VLACCritic

critic = VLACCritic.from_pretrained("InternRobotics/VLAC")

env = YourRobotEnv()
replay_buffer = ReplayBuffer()

for episode in range(200):
    obs_history = []
    obs, _ = env.reset()
    
    for step in range(max_steps):
        action = policy.act(obs)
        next_obs, _, done, _, _ = env.step(action)
        obs_history.append((obs, next_obs))
        
        # VLAC computes reward — no manual reward function needed
        if len(obs_history) >= 2:
            reward_info = critic.web_trajectory_critic(
                images=[obs_history[-2][0]["image"], obs_history[-1][0]["image"]],
                task_description=env.task_description
            )
            reward = reward_info["progress_deltas"][-1]
        else:
            reward = 0.0
        
        replay_buffer.add(obs, action, reward, next_obs, done)
        obs = next_obs
        
        if done:
            break
    
    # Update policy every N episodes
    if episode % 10 == 0:
        policy.update(replay_buffer.sample(batch_size=64))

Experimental Results

VLAC was evaluated on four real-world manipulation tasks using a robot arm:

With 8 robots in parallel, each robot needed only 64 episodes (total 512 episodes, but wall-clock time drops 8x) to reach 80% success — versus 325 episodes for a single-robot setup.

Comparison Against Baselines

• Hand-crafted dense reward: Requires 3–5 engineer-days to design and tune per task. VLAC: zero new engineering per task.
• Sparse reward only (binary done signal): Converges ~4x slower, or fails to converge within the 200-episode budget.
• One-shot transfer: VLAC can evaluate entirely new tasks (unseen during training) from a single demo video — no retraining needed.

Implementation Pitfalls

1. Camera stability matters more than you think

VLAC infers progress from visual change. Shaky cameras, sudden lighting changes, or the robot arm blocking the object of interest all add noise to the reward signal. Fix the camera position and use stable lighting.

2. Keep task descriptions consistent

Use exactly the same phrasing throughout an episode. "Pick up the bowl" and "grab the bowl and move it" are semantically different tasks from VLAC's perspective. Define the task string once and never mutate it mid-episode.

3. Human-in-the-loop operators need training too

The three-tier HiL protocol is not a "hand it to anyone" setup. Operators need to understand when to intervene — intervening at the wrong moment can teach the policy wrong behaviors.

4. VLAC-2B vs VLAC-8B

For real-time inference on edge hardware (Jetson AGX Orin), use VLAC-2B. For offline data refinement (filtering replay buffers for quality), use VLAC-8B for higher accuracy.

5. Negative progress delta is a feature, not a bug

Many users are surprised by negative rewards mid-episode. This is intentional — the critic penalizes the robot for regressing, helping the policy learn to avoid actions that undo previous progress.

Why VLAC Matters for Robotics Practitioners

VLAC demonstrates that the real bottleneck of real-world robot RL is not the RL algorithm itself — it is high-quality reward signals. By learning rewards from data rather than hand-crafting them, combined with a structured human-in-the-loop protocol, VLAC moves real-world RL from "lab-only experiment" to a deployable production workflow achievable within 200 episodes.

Three practical takeaways:

1. Reduce setup time from weeks to days — no per-task reward engineering
2. Transfer to new tasks with a single demo video — no retraining
3. Scale linearly with robot count — double the robots, halve the wall-clock time

Paper: arXiv:2509.15937 | GitHub: InternRobotics/VLAC | Model: HuggingFace VLAC

Related Posts

• Force-VLA RL: Value-Calibrated Warmup for Robot Manipulation
• GigaBrain-0: VLA with World Model and RL for Manipulation
• EXPO-FT Guide: Online RL for Pi0 in Just 19 Minutes