Qwen-VLA: Train Multi-Task Robot Manipulation on LIBERO to 97.9%

Imagine you need to build an engineering team. You have two choices: hire 10 specialists who each do exactly one thing, or hire 3 generalists who can handle everything. In AI robotics, most labs have been going the first route — a separate VLA for each task, a separate policy for each robot arm, a fresh fine-tune for every benchmark.

Qwen-VLA from Alibaba's Qwen team asks a different question: Can a single model learn all of it? The answer turns out to be yes — and it achieves 97.9% on LIBERO while simultaneously handling navigation, real-world manipulation, and egocentric action modeling with a single set of weights.

The Problem: Specialist Hell in VLA

Before Qwen-VLA, the VLA landscape looked like this: OpenVLA for simple manipulation, pi0 for ALOHA bimanual, GR00T N1 for Unitree G1, yet another model for navigation. Each model is good at exactly what it was trained on, but transfers poorly to anything else.

The deeper problem is data efficiency: every specialist must re-learn visual grounding, spatial reasoning, and action generation from scratch. A generalist can share representations across tasks — something the human brain does effortlessly but AI robotics has struggled with.

Qwen-VLA addresses this by casting manipulation, navigation, egocentric action, and trajectory prediction into a unified action-and-trajectory space, steered by embodiment-aware text prompts.

Architecture: Qwen3.5-4B + 1.15B DiT Decoder

Qwen-VLA has two main components:

1. VLM Backbone: Qwen3.5-4B

The backbone is Qwen3.5-4B — Alibaba's 4-billion-parameter vision-language model. Pre-trained on massive text and image data, it already understands visual grounding, spatial reasoning, and natural language deeply.

Its role: take camera frames + language instruction → encode into rich semantic token representations.

2. Action Decoder: 1.15B DiT Flow-Matching

The second component is a 1.15B DiT (Diffusion Transformer) action decoder using a flow-matching objective. This is where actual actions are generated.

Flow-matching differs from standard diffusion: instead of learning to denoise from pure noise, it learns a velocity field mapping directly from a noise distribution to an action distribution. The result: inference requires only a few Euler integration steps — fast enough for real-time robot control.

Input: Camera frames + Language instruction + Embodiment prompt
         ↓
  Qwen3.5-4B VLM Backbone
  (Visual encoding + Language understanding)
         ↓
  Token representations
         ↓
  1.15B DiT Flow-Matching Action Decoder
  (Few Euler steps)
         ↓
  Continuous action trajectory (joint angles, end-effector poses...)

3. Embodiment-Aware Prompt Conditioning

This is the most important insight: rather than separate per-platform output heads, Qwen-VLA uses robot-specific text prompts to condition action generation.

For example:

"You are controlling a WidowX single-arm robot with 6 DOF. 
 Action space: [joint1, joint2, joint3, joint4, joint5, joint6, gripper]"

vs.

"You are controlling an ALOHA dual-arm robot. 
 Action space: [left_arm_6dof, left_gripper, right_arm_6dof, right_gripper]"

Just swap the prompt and the model switches embodiment. 11 different robot embodiments share a single set of weights.

Pre-Training Data: 74% Manipulation, 7.5% Navigation

Qwen-VLA is trained on a massive dataset with this composition:

Notably, 6% egocentric human data — the model learns from videos of humans working (using MANO hand models + 10D PCA eigengrasps to extract hand trajectories), requiring no robot data for this portion.

Also important: per-dataset quantile normalization for action scaling — each dataset has different action ranges, and normalization ensures the DiT decoder isn't biased toward any single dataset.

4-Stage Training: From Text to Real World

Qwen-VLA's progressive training curriculum is what separates it from a simple pretrain-finetune approach:

Stage I: Text-to-Action (T2A) Pre-training

In the first stage, the model receives only language and embodiment prompts — no vision. The task: learn language-indexed action priors — what a "pick up the cup" trajectory should structurally look like.

Paper finding: the optimal mixture is 20% synthetic + 80% real data for downstream SFT performance.

Stage II: Continued Pre-training (CPT)

Vision is introduced in stage two. The model learns joint backbone + DiT training with visual input, using heterogeneous real, simulated, and synthetic data.

This is where the model learns to ground language in visual space — a critical skill for real-world manipulation.

Stage III: Supervised Fine-Tuning (SFT)

SFT on target task demonstrations. For LIBERO, this is where the model learns specific benchmark tasks.

Key: balanced data mixtures across embodiments prevent any single platform from dominating.

Stage IV: Reinforcement Learning (RL)

The final stage uses PPO/GAE optimization for closed-loop task success. RL here focuses on single-environment policy improvement while retaining mild cross-domain transfer benefits.

Installation and Setup

# Clone repo
git clone https://github.com/QwenLM/Qwen-VLA.git
cd Qwen-VLA

# Create conda environment (Python 3.10+ required)
conda create -n qwen-vla python=3.10
conda activate qwen-vla

# Install PyTorch with CUDA 12.1
pip install torch==2.3.0 torchvision==0.18.0 --index-url https://download.pytorch.org/whl/cu121

# Install transformers (4.57.0+ required for Qwen3.5 architecture)
pip install transformers>=4.57.0

# Install core dependencies
pip install accelerate einops timm
pip install flash-attn --no-build-isolation  # Flash Attention 2 for speed

# Install Qwen utilities
pip install qwen-vl-utils[decord]==0.0.8

Hardware requirements:

• Training: minimum 4× A100 80GB (recommended 8× A100 or H100)
• Inference: RTX 4090 (24GB) or A10G (24GB) sufficient
• Disk: ~50GB for weights + dataset

Download Model Weights

from huggingface_hub import snapshot_download

# Download Qwen-VLA-Instruct (full model)
snapshot_download(
    repo_id="Qwen/Qwen-VLA-Instruct",
    local_dir="./checkpoints/qwen-vla-instruct"
)

# Or via CLI
# huggingface-cli download Qwen/Qwen-VLA-Instruct

Running Inference on LIBERO

Basic inference example with LIBERO environment:

import torch
from transformers import AutoProcessor, AutoModelForCausalLM

# Load model and processor
model_path = "./checkpoints/qwen-vla-instruct"
processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True
)

# Embodiment prompt for LIBERO (Franka Panda)
EMBODIMENT_PROMPT = """You are controlling a Franka Panda robot arm.
Action space: [x, y, z, roll, pitch, yaw, gripper] in end-effector space.
"""

def get_action(observation, task_instruction, images):
    """
    observation: dict with camera frames
    task_instruction: str, e.g. "Pick up the red cup and place it in the bowl"
    images: list of PIL Images from cameras
    """
    messages = [
        {"role": "system", "content": EMBODIMENT_PROMPT},
        {
            "role": "user", 
            "content": [
                *[{"type": "image", "image": img} for img in images],
                {"type": "text", "text": task_instruction}
            ]
        }
    ]
    
    text = processor.apply_chat_template(
        messages, tokenize=False, add_generation_prompt=True
    )
    inputs = processor(
        text=[text], images=images, return_tensors="pt"
    ).to(model.device, torch.bfloat16)
    
    with torch.no_grad():
        action = model.generate_action(**inputs)
    
    return action.cpu().numpy()

# Use in LIBERO eval loop
from libero.libero import benchmark

benchmark_dict = benchmark.get_benchmark_dict()
task_suite = benchmark_dict["libero_spatial"]()
task = task_suite.get_task(0)
env = task_suite.get_task_init_states(0)

obs = env.reset()
for step in range(300):
    agentview_img = obs["agentview_image"]
    wrist_img = obs["robot0_eye_in_hand_image"]
    
    images = [agentview_img, wrist_img]
    action = get_action(obs, task.language, images)
    
    obs, reward, done, info = env.step(action)
    if done:
        print(f"Task completed at step {step}!")
        break

Fine-Tuning on Custom Data

To fine-tune Qwen-VLA on your own robot data, format it as follows:

import json
from pathlib import Path

def prepare_training_data(episodes: list, output_dir: str):
    """
    episodes: list of dicts, each representing one episode:
    {
        "observations": [...],  # list of frames
        "actions": [...],       # list of action vectors
        "task": "Pick up the blue block",
        "embodiment": "franka_panda"
    }
    """
    output_path = Path(output_dir)
    output_path.mkdir(parents=True, exist_ok=True)
    
    for i, ep in enumerate(episodes):
        ep_data = {
            "task_instruction": ep["task"],
            "embodiment_prompt": get_embodiment_prompt(ep["embodiment"]),
            "frames": ep["observations"],
            "actions": ep["actions"],
            "action_dim": len(ep["actions"][0])
        }
        
        with open(output_path / f"episode_{i:05d}.json", "w") as f:
            json.dump(ep_data, f)

def get_embodiment_prompt(robot_name: str) -> str:
    prompts = {
        "franka_panda": "You are controlling a Franka Panda 7-DOF robot arm...",
        "widowx": "You are controlling a WidowX 6-DOF robot arm...",
        "aloha": "You are controlling an ALOHA dual-arm bimanual robot..."
    }
    return prompts.get(robot_name, "")

Launch Training

# SFT on LIBERO with 4 GPUs
torchrun --nproc_per_node=4 train.py \
    --model_name_or_path ./checkpoints/qwen-vla-instruct \
    --dataset_path ./data/libero_spatial \
    --output_dir ./checkpoints/qwen-vla-libero-ft \
    --embodiment franka_panda \
    --num_train_epochs 50 \
    --per_device_train_batch_size 4 \
    --gradient_accumulation_steps 4 \
    --learning_rate 2e-5 \
    --warmup_ratio 0.05 \
    --bf16 True \
    --save_strategy epoch \
    --save_total_limit 3

# RL fine-tuning after SFT (requires LIBERO env)
python train_rl.py \
    --checkpoint ./checkpoints/qwen-vla-libero-ft \
    --env libero_spatial \
    --num_envs 8 \
    --rl_algo ppo \
    --total_timesteps 500000

Demo and Architecture

Architecture overview — a single model for all tasks and embodiments:

Source: QwenLM/Qwen-VLA

Results: Simulation Benchmarks

Qwen-VLA is trained once on all embodiments jointly and evaluated without per-benchmark adaptation:

The model also handles navigation: R2R OSR 69.0%, RxR SR 59.6% — while specialist navigation models typically cannot handle manipulation.

Out-of-Distribution Generalization

DOMINO is zero-shot evaluation with moving objects — no dynamic object training data, yet the model achieves 26.6%.

Real-World ALOHA: Beating Specialists at Their Own Game

The most impressive results come from the ALOHA bimanual platform with 6 real tasks:

In-Domain Performance (% success):

OOD Performance — unseen colors, positions, object instances:

Pre-training lifts OOD average from 36.2% (no pre-training) to 76.9% — nearly doubling it. Meanwhile π₀.₅, one of the strongest specialist baselines, reaches only 41.5% OOD.

Why Pre-Training Changes Everything

The ALOHA results reveal a striking pattern: without pre-training, Qwen-VLA only reaches 48.5% in-domain — worse than the π₀.₅ specialist. With pre-training: 83.6% in-domain, 76.9% OOD.

Why? Large-scale embodied pre-training teaches the model transferable representations — how to recognize objects, understand spatial relationships, and map language to physical space. These skills transfer robustly to unseen conditions (different colors, positions, paraphrased instructions).

This is the strongest argument for the generalist VLA direction: specialists learn from less data but are brittle; generalists learn from more and are robust. The key isn't just the architecture — it's the scale and diversity of pre-training data.

Related Posts

• LabVLA: Using Qwen3-VL + LeRobot to Train VLA for Lab Robots
• TGRPO: Fine-Tuning VLA on LIBERO with Trajectory GRPO
• OpenVLA — Deep Dive Architecture and Training