SimpleVLA-RL: When VLA Models Learn From Their Own Failures
Here is a frustrating paradox in modern robot manipulation: you can spend months collecting 10,000 teleoperation demonstrations, train a massive VLA model, and still find yourself stuck at 70–80% task success. Add another 10,000 demos? Success rate might creep up a few percentage points. This is the data ceiling — the performance plateau that every Supervised Fine-Tuning (SFT) approach eventually hits.
SimpleVLA-RL, published at ICLR 2026 by the PRIME-RL team, offers a surprisingly elegant escape: let the robot practice in simulation with binary feedback (success = 1, failure = 0), no reward engineering required. The result? From 91% to 99.1% on LIBERO, and a 430% improvement in data-scarce regimes.
This article is a complete practical guide — from understanding the paper's insight to running the code on your GPU cluster.
Why SFT Is Not Enough
Before diving into code, it is worth understanding the problem SimpleVLA-RL solves.
When you train a VLA model like OpenVLA with SFT, the model learns to imitate expert demonstrations. This works well when the robot encounters exactly the situations seen during training. But three fundamental problems remain:
- Distribution shift: The real robot sees different lighting, slightly different camera angles, objects at slightly different positions — the model fails to generalize
- Compounding errors: Each small mistake accumulates; after 100 steps, small errors compound into complete task failure
- Mode collapse in SFT: The model learns an average of the demonstrations, never learning recovery strategies for when things go wrong
Reinforcement learning solves all three by letting the robot explore strategies that demonstration data never shows — including how to recover from error states.
SimpleVLA-RL Architecture

SimpleVLA-RL is built on three main components:
1. VLA Backbone: OpenVLA-OFT
OpenVLA-OFT is a fine-tuned version of OpenVLA with Optimal Fine-Tuning — optimized for real robot manipulation. This serves as the starting checkpoint before applying RL.
2. RL Engine: veRL
veRL is a high-performance distributed RL framework originally designed for LLM training. PRIME-RL extended veRL with VLA-specific components:
- VLA-specific trajectory sampling: Sample trajectories from multiple simulation environments in parallel, not text tokens
- Multi-environment rendering: Render multiple simulation scenes simultaneously to increase RL data throughput
- Optimized loss computation: Compute policy gradients over VLA model action token sequences
3. Algorithm: GRPO
Instead of the complex PPO, SimpleVLA-RL uses Group Relative Policy Optimization (GRPO) — the same algorithm used in DeepSeek-R1. GRPO compares multiple rollouts from the same state and computes relative rewards:
# Pseudo-code for GRPO reward calculation
for state in batch_states:
rollouts = [policy.sample(state) for _ in range(G)] # G rollouts
rewards = [env.check_success(r) for r in rollouts] # 0 or 1
baseline = mean(rewards)
advantages = [r - baseline for r in rewards] # relative reward
# Update policy to increase probability of good rollouts
Why GRPO instead of PPO? PPO requires a separate value function (critic network), consuming extra memory and compute. GRPO needs no critic — it uses the parallel rollouts themselves as the baseline, making it simpler and better suited to decoder-only VLA architectures.
The "Pushcut" Phenomenon: VLA Discovers New Strategies
One of the most interesting findings in the paper is the pushcut phenomenon: after RL training, the model discovers an approach not present in any demonstration. Instead of grasping directly, the robot sometimes gently pushes an object to adjust its angle, then grasps — a strategy no demo ever taught.
This is strong evidence that RL does not merely improve execution quality; it genuinely expands the robot's strategic space.
Hardware Requirements
SimpleVLA-RL is not a laptop project. Minimum requirements:
| Component | Requirement |
|---|---|
| GPU | 8× NVIDIA A800 80GB (or A100 80GB) |
| CUDA | 12.4+ |
| RAM | 256GB+ |
| Storage | 500GB+ (SFT checkpoint + logs) |
| Python | 3.10 |
If you do not have an in-house GPU cluster, cloud GPU options include Lambda Labs, RunPod, or AWS p4d.24xlarge.
Environment Setup
Step 1: Create conda environment
conda create -n simplevla python=3.10 -y
conda activate simplevla
Step 2: Clone the main repository
git clone https://github.com/PRIME-RL/SimpleVLA-RL.git
cd SimpleVLA-RL
Step 3: Install dependencies
# Install veRL and core dependencies
pip install -e .
# Install OpenVLA-OFT
git clone https://github.com/moojink/openvla-oft.git
cd openvla-oft && pip install -e . && cd ..
Step 4: Install LIBERO (for LIBERO benchmark)
pip install git+https://github.com/Lifelong-Robot-Learning/LIBERO.git
Step 5: Install RoboTwin 2.0 (for RoboTwin benchmark)
# Follow instructions at the RoboTwin 2.0 repository
pip install git+https://github.com/TeleVU-Org/RoboTwin.git
Step 6: Configure WandB
SimpleVLA-RL uses WandB for experiment tracking. Open align.json and add your API key:
{
"wandb_api_key": "YOUR_WANDB_API_KEY_HERE"
}
Stage 1: SFT Cold-Start
RL cannot start from a random model — it needs a starting point with basic task competence. This is the role of SFT cold-start.
Download SFT checkpoints
PRIME-RL provides pre-built SFT checkpoints on HuggingFace:
# Checkpoint for LIBERO with 10 trajectories per task
huggingface-cli download PRIME-RL/SimpleVLA-RL-Collection \
--include "libero-10-traj1-sft/*" \
--local-dir ./checkpoints/
# Checkpoint for LIBERO with all trajectories
huggingface-cli download PRIME-RL/SimpleVLA-RL-Collection \
--include "libero-10-trajall-sft/*" \
--local-dir ./checkpoints/
Why "1 trajectory per task" matters
The paper reports results with only 1 trajectory per task for cold-start SFT — an extremely data-scarce regime. SFT alone achieves just 17.3% success. After RL, this jumps to 91.7% — the most compelling evidence of RL's power to break through the data ceiling.
| Regime | SFT | RL |
|---|---|---|
| 1 trajectory/task | 17.3% | 91.7% |
| 100 demos | 7.3% | 25.4% |
| 1000 demos | 28.2% | 50.4% |
| Full LIBERO | 91.0% | 99.1% |
Stage 2: RL Training on LIBERO
Configure the training script
Open examples/run_openvla_oft_rl_libero.sh and adjust:
# Number of nodes (1 node = 8 GPUs)
export NUM_NODES=1
# Path to SFT checkpoint
export ALIGN_PATH="./checkpoints/libero-10-traj1-sft"
# Number of parallel environments (increase for higher throughput)
export N_ENVS=8
# LIBERO task suite: libero_spatial, libero_object, libero_goal, libero_long
export TASK_SUITE="libero_long"
Run training
bash examples/run_openvla_oft_rl_libero.sh
This script will:
- Launch veRL with distributed training
- Start 8 LIBERO simulation environments in parallel
- Each rollout: robot attempts the task, receives reward 1.0 (success) or 0.0 (failure)
- Compute GRPO loss and update the policy
- Repeat for ~1000 iterations
Monitoring
Open wandb.ai to view real-time metrics:
train/success_rate: Current success ratetrain/policy_loss: Policy gradient losstrain/entropy: Policy entropy (too low = collapsed policy)
Stage 3: RL Training on RoboTwin 2.0
RoboTwin 2.0 is harder than LIBERO, with more complex manipulation tasks and greater environment diversity.
export ALIGN_PATH="./checkpoints/robottwin-sft"
export N_ENVS=8
bash examples/run_openvla_oft_rl_twin2.sh
Results on RoboTwin 2.0
According to the paper and independent analysis (dVLA-RL, 2025):
| Model | Average Success Rate |
|---|---|
| OpenVLA-OFT (SFT only) | 34.2% |
| SimpleVLA-RL | 68.7% |
| π0 | Below SimpleVLA-RL |
Surpassing π0 — one of the strongest VLA models available — is a remarkable result.
Inference: Running the Trained Model
After training, checkpoints are saved at outputs/<experiment_name>/. For inference:
from openvla_oft import OpenVLAOFT
import numpy as np
# Load trained model
model = OpenVLAOFT.from_pretrained(
"outputs/simplevla_rl_libero_long/checkpoint-1000"
)
model.eval()
# Inference loop
obs = env.reset()
for step in range(max_steps):
action = model.predict_action(
image=obs["image"],
instruction=obs["instruction"]
)
obs, reward, done, _ = env.step(action)
if done:
print(f"Task completed in {step+1} steps!")
break
Sim-to-Real Transfer
Notably, SimpleVLA-RL is trained entirely in simulation yet transfers to real robots:
| Environment | SFT | SimpleVLA-RL |
|---|---|---|
| Simulation (RoboTwin 2.0) | 34.2% | 68.7% |
| Real robot (sim-only training) | 17.5% | 38.5% |
The sim-to-real gap remains (68.7% → 38.5%), but RL significantly improves real robot performance without any additional real-world data.

Common Errors and Fixes
Error 1: CUDA out of memory
RuntimeError: CUDA out of memory. Tried to allocate X GiB
Reduce N_ENVS or batch_size in the script. With 8× A100 40GB (instead of 80GB), reduce settings to roughly 60–70% of defaults.
Error 2: LIBERO import error
ModuleNotFoundError: No module named 'robosuite'
LIBERO depends on robosuite: pip install robosuite
Error 3: Reward always zero
Check that the task setup is correct — the robot may not be seeing the target object due to camera misconfiguration. Run python examples/visualize_env.py to inspect.
Error 4: Training not converging after 500 iterations
Try reducing learning rate: --lr 1e-6 instead of the default 5e-6. VLA models are sensitive to learning rate.
Limitations and Future Work
Current limitations worth knowing:
- Only supports OpenVLA/OpenVLA-OFT: Support for π0 and other VLA models is in development (pending feature on GitHub)
- High GPU requirement: 8× A800 80GB is not universally accessible; the community is exploring 4× A100 40GB configurations
- Binary reward only: No dense reward — learning can be slow for complex tasks. Community forks add dense reward but are not officially supported
Key Takeaway
If there is one thing to take away from this guide: the simplest possible reward (0/1) is sufficient to surpass large-scale SFT, given enough compute for simulation rollouts.
This opens a new paradigm: instead of spending resources collecting demonstration data, invest in simulation infrastructure and RL compute to let the model improve itself.



