VnRoboVnRobo
AboutPricingBlogContact
🇻🇳VISign InStart learning
🇻🇳VI
VnRobo logoVnRobo logo

Learn Physical AI and experiment in your browser. Connect your fleet when you have real robots.

Product

  • Features
  • Pricing
  • Knowledge Base
  • Services

Company

  • About Us
  • Blog
  • Contact

Legal

  • Privacy Policy
  • Terms of Service

© 2026 VnRobo. All rights reserved.

Made with♥in Vietnam
VnRoboVnRobo
AboutPricingBlogContact
🇻🇳VISign InStart learning
🇻🇳VI
  1. Home
  2. Blog
  3. SimpleVLA-RL Practical Guide: Scale VLA with RL
wholebody-vlavlareinforcement-learningmanipulationopenvlaliberotrainingimitation-learning

SimpleVLA-RL Practical Guide: Scale VLA with RL

End-to-end guide to SimpleVLA-RL (ICLR 2026) — use binary 0/1 rewards to scale VLA manipulation by 430%, with step-by-step code from install to real robot.

Nguyễn Anh TuấnNovember 17, 20269 min read
SimpleVLA-RL Practical Guide: Scale VLA with RL

SimpleVLA-RL: When VLA Models Learn From Their Own Failures

Here is a frustrating paradox in modern robot manipulation: you can spend months collecting 10,000 teleoperation demonstrations, train a massive VLA model, and still find yourself stuck at 70–80% task success. Add another 10,000 demos? Success rate might creep up a few percentage points. This is the data ceiling — the performance plateau that every Supervised Fine-Tuning (SFT) approach eventually hits.

SimpleVLA-RL, published at ICLR 2026 by the PRIME-RL team, offers a surprisingly elegant escape: let the robot practice in simulation with binary feedback (success = 1, failure = 0), no reward engineering required. The result? From 91% to 99.1% on LIBERO, and a 430% improvement in data-scarce regimes.

This article is a complete practical guide — from understanding the paper's insight to running the code on your GPU cluster.


Why SFT Is Not Enough

Before diving into code, it is worth understanding the problem SimpleVLA-RL solves.

When you train a VLA model like OpenVLA with SFT, the model learns to imitate expert demonstrations. This works well when the robot encounters exactly the situations seen during training. But three fundamental problems remain:

  1. Distribution shift: The real robot sees different lighting, slightly different camera angles, objects at slightly different positions — the model fails to generalize
  2. Compounding errors: Each small mistake accumulates; after 100 steps, small errors compound into complete task failure
  3. Mode collapse in SFT: The model learns an average of the demonstrations, never learning recovery strategies for when things go wrong

Reinforcement learning solves all three by letting the robot explore strategies that demonstration data never shows — including how to recover from error states.


SimpleVLA-RL Architecture

SimpleVLA-RL architecture overview — source: PRIME-RL/SimpleVLA-RL
SimpleVLA-RL architecture overview — source: PRIME-RL/SimpleVLA-RL

SimpleVLA-RL is built on three main components:

1. VLA Backbone: OpenVLA-OFT

OpenVLA-OFT is a fine-tuned version of OpenVLA with Optimal Fine-Tuning — optimized for real robot manipulation. This serves as the starting checkpoint before applying RL.

2. RL Engine: veRL

veRL is a high-performance distributed RL framework originally designed for LLM training. PRIME-RL extended veRL with VLA-specific components:

  • VLA-specific trajectory sampling: Sample trajectories from multiple simulation environments in parallel, not text tokens
  • Multi-environment rendering: Render multiple simulation scenes simultaneously to increase RL data throughput
  • Optimized loss computation: Compute policy gradients over VLA model action token sequences

3. Algorithm: GRPO

Instead of the complex PPO, SimpleVLA-RL uses Group Relative Policy Optimization (GRPO) — the same algorithm used in DeepSeek-R1. GRPO compares multiple rollouts from the same state and computes relative rewards:

python
# Pseudo-code for GRPO reward calculation
for state in batch_states:
    rollouts = [policy.sample(state) for _ in range(G)]  # G rollouts
    rewards = [env.check_success(r) for r in rollouts]   # 0 or 1
    baseline = mean(rewards)
    advantages = [r - baseline for r in rewards]          # relative reward
    # Update policy to increase probability of good rollouts

Why GRPO instead of PPO? PPO requires a separate value function (critic network), consuming extra memory and compute. GRPO needs no critic — it uses the parallel rollouts themselves as the baseline, making it simpler and better suited to decoder-only VLA architectures.


The "Pushcut" Phenomenon: VLA Discovers New Strategies

One of the most interesting findings in the paper is the pushcut phenomenon: after RL training, the model discovers an approach not present in any demonstration. Instead of grasping directly, the robot sometimes gently pushes an object to adjust its angle, then grasps — a strategy no demo ever taught.

This is strong evidence that RL does not merely improve execution quality; it genuinely expands the robot's strategic space.


Hardware Requirements

SimpleVLA-RL is not a laptop project. Minimum requirements:

Component Requirement
GPU 8× NVIDIA A800 80GB (or A100 80GB)
CUDA 12.4+
RAM 256GB+
Storage 500GB+ (SFT checkpoint + logs)
Python 3.10

If you do not have an in-house GPU cluster, cloud GPU options include Lambda Labs, RunPod, or AWS p4d.24xlarge.

Tool recommendations

VLA train/deploy stack

Train on cloud/workstation, then deploy optimized models to Jetson or the robot computer.

Cloud GPU for VLA / policy training Use for imitation learning, diffusion policies, RL, and robotics model fine-tuning. View cloud GPU → NVIDIA Jetson Orin NX / Orin Nano Edge deployment hardware for perception, logging, and optimized inference. View Jetson → Hugging Face / robotics dataset hosting Host datasets, checkpoints, and model cards for cleaner LeRobot/VLA workflows. View platform →

Environment Setup

Step 1: Create conda environment

bash
conda create -n simplevla python=3.10 -y
conda activate simplevla

Step 2: Clone the main repository

bash
git clone https://github.com/PRIME-RL/SimpleVLA-RL.git
cd SimpleVLA-RL

Step 3: Install dependencies

bash
# Install veRL and core dependencies
pip install -e .

# Install OpenVLA-OFT
git clone https://github.com/moojink/openvla-oft.git
cd openvla-oft && pip install -e . && cd ..

Step 4: Install LIBERO (for LIBERO benchmark)

bash
pip install git+https://github.com/Lifelong-Robot-Learning/LIBERO.git

Step 5: Install RoboTwin 2.0 (for RoboTwin benchmark)

bash
# Follow instructions at the RoboTwin 2.0 repository
pip install git+https://github.com/TeleVU-Org/RoboTwin.git

Step 6: Configure WandB

SimpleVLA-RL uses WandB for experiment tracking. Open align.json and add your API key:

json
{
    "wandb_api_key": "YOUR_WANDB_API_KEY_HERE"
}

Stage 1: SFT Cold-Start

RL cannot start from a random model — it needs a starting point with basic task competence. This is the role of SFT cold-start.

Download SFT checkpoints

PRIME-RL provides pre-built SFT checkpoints on HuggingFace:

bash
# Checkpoint for LIBERO with 10 trajectories per task
huggingface-cli download PRIME-RL/SimpleVLA-RL-Collection \
    --include "libero-10-traj1-sft/*" \
    --local-dir ./checkpoints/

# Checkpoint for LIBERO with all trajectories
huggingface-cli download PRIME-RL/SimpleVLA-RL-Collection \
    --include "libero-10-trajall-sft/*" \
    --local-dir ./checkpoints/

Why "1 trajectory per task" matters

The paper reports results with only 1 trajectory per task for cold-start SFT — an extremely data-scarce regime. SFT alone achieves just 17.3% success. After RL, this jumps to 91.7% — the most compelling evidence of RL's power to break through the data ceiling.

Regime SFT RL
1 trajectory/task 17.3% 91.7%
100 demos 7.3% 25.4%
1000 demos 28.2% 50.4%
Full LIBERO 91.0% 99.1%

Stage 2: RL Training on LIBERO

Configure the training script

Open examples/run_openvla_oft_rl_libero.sh and adjust:

bash
# Number of nodes (1 node = 8 GPUs)
export NUM_NODES=1

# Path to SFT checkpoint
export ALIGN_PATH="./checkpoints/libero-10-traj1-sft"

# Number of parallel environments (increase for higher throughput)
export N_ENVS=8

# LIBERO task suite: libero_spatial, libero_object, libero_goal, libero_long
export TASK_SUITE="libero_long"

Run training

bash
bash examples/run_openvla_oft_rl_libero.sh

This script will:

  1. Launch veRL with distributed training
  2. Start 8 LIBERO simulation environments in parallel
  3. Each rollout: robot attempts the task, receives reward 1.0 (success) or 0.0 (failure)
  4. Compute GRPO loss and update the policy
  5. Repeat for ~1000 iterations

Monitoring

Open wandb.ai to view real-time metrics:

  • train/success_rate: Current success rate
  • train/policy_loss: Policy gradient loss
  • train/entropy: Policy entropy (too low = collapsed policy)

Stage 3: RL Training on RoboTwin 2.0

RoboTwin 2.0 is harder than LIBERO, with more complex manipulation tasks and greater environment diversity.

bash
export ALIGN_PATH="./checkpoints/robottwin-sft"
export N_ENVS=8

bash examples/run_openvla_oft_rl_twin2.sh

Results on RoboTwin 2.0

According to the paper and independent analysis (dVLA-RL, 2025):

Model Average Success Rate
OpenVLA-OFT (SFT only) 34.2%
SimpleVLA-RL 68.7%
π0 Below SimpleVLA-RL

Surpassing π0 — one of the strongest VLA models available — is a remarkable result.


Inference: Running the Trained Model

After training, checkpoints are saved at outputs/<experiment_name>/. For inference:

python
from openvla_oft import OpenVLAOFT
import numpy as np

# Load trained model
model = OpenVLAOFT.from_pretrained(
    "outputs/simplevla_rl_libero_long/checkpoint-1000"
)
model.eval()

# Inference loop
obs = env.reset()
for step in range(max_steps):
    action = model.predict_action(
        image=obs["image"],
        instruction=obs["instruction"]
    )
    obs, reward, done, _ = env.step(action)
    if done:
        print(f"Task completed in {step+1} steps!")
        break

Sim-to-Real Transfer

Notably, SimpleVLA-RL is trained entirely in simulation yet transfers to real robots:

Environment SFT SimpleVLA-RL
Simulation (RoboTwin 2.0) 34.2% 68.7%
Real robot (sim-only training) 17.5% 38.5%

The sim-to-real gap remains (68.7% → 38.5%), but RL significantly improves real robot performance without any additional real-world data.

SimpleVLA-RL teaser results — source: PRIME-RL/SimpleVLA-RL
SimpleVLA-RL teaser results — source: PRIME-RL/SimpleVLA-RL


Common Errors and Fixes

Error 1: CUDA out of memory

code
RuntimeError: CUDA out of memory. Tried to allocate X GiB

Reduce N_ENVS or batch_size in the script. With 8× A100 40GB (instead of 80GB), reduce settings to roughly 60–70% of defaults.

Error 2: LIBERO import error

code
ModuleNotFoundError: No module named 'robosuite'

LIBERO depends on robosuite: pip install robosuite

Error 3: Reward always zero Check that the task setup is correct — the robot may not be seeing the target object due to camera misconfiguration. Run python examples/visualize_env.py to inspect.

Error 4: Training not converging after 500 iterations Try reducing learning rate: --lr 1e-6 instead of the default 5e-6. VLA models are sensitive to learning rate.


Limitations and Future Work

Current limitations worth knowing:

  1. Only supports OpenVLA/OpenVLA-OFT: Support for π0 and other VLA models is in development (pending feature on GitHub)
  2. High GPU requirement: 8× A800 80GB is not universally accessible; the community is exploring 4× A100 40GB configurations
  3. Binary reward only: No dense reward — learning can be slow for complex tasks. Community forks add dense reward but are not officially supported

Key Takeaway

If there is one thing to take away from this guide: the simplest possible reward (0/1) is sufficient to surpass large-scale SFT, given enough compute for simulation rollouts.

This opens a new paradigm: instead of spending resources collecting demonstration data, invest in simulation infrastructure and RL compute to let the model improve itself.


Related Posts

  • SimpleVLA-RL (1): Overview and Core Idea
  • SimpleVLA-RL (2): Architecture and GRPO Algorithm
  • RL-VLA³: Asynchronous RL Framework for VLA Training
NT

Nguyễn Anh Tuấn

Robotics & AI Engineer. Building VnRobo — sharing knowledge about robot learning, VLA models, and automation.

Explore VnRobo

Fleet MonitoringROS 2 IntegrationAMR Solutions
← Previous
SLA: Continual Pre-training π0.5 with 3D Spatial Supervision (+24%)

Related Posts

Research
TGRPO: Fine-tuning VLA with Trajectory GRPO and LLM Dense Reward
vlagrporeinforcement-learning
wholebody-vla

TGRPO: Fine-tuning VLA with Trajectory GRPO and LLM Dense Reward

TGRPO combines LLM-generated dense rewards with dual-level advantage estimation (step + trajectory) to fine-tune OpenVLA-7B on LIBERO, reaching 91% success rate — beating SFT by 4.6%.

7/1/202610 min read
NT
Tutorial
RISE: Hands-on Training Pipeline Guide
riseworld-modelvla
wholebody-vla

RISE: Hands-on Training Pipeline Guide

Step-by-step walkthrough of the RISE training pipeline: environment setup, LeRobot data preparation, offline policy training, dynamics model, and online self-improvement loop on 4–8 GPUs.

6/13/202612 min read
NT
Deep Dive
VLA-RFT: RL Fine-Tuning VLAs in World Simulators
vlavla-rftreinforcement-learning
wholebody-vla

VLA-RFT: RL Fine-Tuning VLAs in World Simulators

A practical deep dive into VLA-RFT: world-model rollouts, verified rewards, GRPO fine-tuning, GitHub code, LIBERO training and results.

6/3/202614 min read
NT
VnRobo logoVnRobo logo

Learn Physical AI and experiment in your browser. Connect your fleet when you have real robots.

Product

  • Features
  • Pricing
  • Knowledge Base
  • Services

Company

  • About Us
  • Blog
  • Contact

Legal

  • Privacy Policy
  • Terms of Service

© 2026 VnRobo. All rights reserved.

Made with♥in Vietnam