InstructVLA: VLA Instruction Tuning Beats OpenVLA by 96%

Imagine training a master mechanic. At first, he reads thousands of technical books and can explain every concept with precision. Then you have him work on the shop floor for six months — and slowly, he forgets all the theory, working purely by muscle memory. That is exactly the problem most VLA models face today.

InstructVLA (ICLR 2026), from researchers at USTC, Zhejiang University, and the Shanghai Artificial Intelligence Laboratory, solves this elegantly. Instead of forcing a model to choose between "understanding language" and "performing manipulation," they design a training pipeline that achieves both — and the results beat OpenVLA by 96% on the new SimplerEnv-Instruct benchmark.

The Problem: VLA Models "Forget" Language After Manipulation Training

Before InstructVLA, most VLA models followed a straightforward recipe: take a powerful Vision-Language Model (VLM) like LLaMA or InternVL, then fine-tune it on robot manipulation data to generate actions. Sounds reasonable.

The problem: manipulation fine-tuning overwrites the language knowledge acquired during pretraining. The model becomes excellent at moving the robot arm, but loses its ability to parse complex instructions. Tell it "pick up the red cup but avoid the one near the stove" — the model sees "red cup," starts reaching, and ignores the "avoid" logic entirely.

This isn't a minor issue. In real deployments, users speak to robots using natural language — long sentences, contextual references, sometimes indirect commands ("the top shelf looks full" instead of "move items from the top shelf"). If the robot can't handle this, the entire Human-Robot Interaction (HRI) layer collapses.

The Solution: Vision-Language-Action Instruction Tuning (VLA-IT)

InstructVLA proposes a new training framework called VLA-IT (Vision-Language-Action Instruction Tuning) built on three pillars:

Pillar 1: Preserving VLM Knowledge with MoE LoRA

Rather than fine-tuning the entire VLM backbone (which causes catastrophic forgetting), InstructVLA uses Mixture-of-Experts LoRA: each layer in the LLM backbone contains multiple LoRA modules acting as independent "experts." A scalar gate coefficient (λᵢ) dynamically weights each expert's contribution per task.

Think of it like a panel of specialists being called into a meeting: whoever is most relevant to the current question gets the microphone, while others still contribute but more quietly.

Pillar 2: Latent Action Queries — Bridging Language and Action

Rather than directly decoding from VLM hidden states to action vectors, InstructVLA inserts a layer of N learnable action queries in between. These queries attend to VLM hidden states, extract task-relevant representations, and pass them down to the action decoder.

The VLM hidden states contain rich language and visual information — far more than the action decoder needs. Action queries act as an intelligent filter, extracting only the "motion-relevant" signals.

Pillar 3: Flow Matching Action Decoder

InstructVLA's action decoder uses flow matching (similar to π0) — a generative modeling technique that outperforms diffusion in inference speed. The decoder consists of:

• DINOv2 as a secondary vision encoder (224×224 resolution)
• Feature-wise Linear Modulation (FiLM) to inject VLM information into the action decoder
• A 12-layer transformer with hidden size 768

The full 3-step action generation process:

1. VLM (Eagle2-2B, 448×448) reads image + instruction, produces hidden states
2. Action queries filter hidden states → latent action embedding
3. Flow matching decoder generates a continuous action trajectory

The VLA-IT Dataset: 650K Samples with 4 Annotation Types

One of the paper's most important contributions is the VLA-IT dataset: 650,000 samples generated from RT-1 Fractal and Bridge Datasets, annotated with GPT-4o's assistance.

GPT-4o receives 3 frames from each robot episode along with the original instruction, then generates 4 annotation types:

These four annotation types create enormous linguistic diversity. The model must learn to "read between the lines" rather than keyword-match.

During training, standard VLM data and manipulation data are mixed at a 1:7 ratio — one multimodal batch for every seven manipulation batches. This ratio was optimized through ablation studies.

SimplerEnv-Instruct: A New Benchmark for Instruction Following

The paper also introduces SimplerEnv-Instruct — a new benchmark with 80 zero-shot manipulation tasks and 1,100 trials, specifically designed to evaluate complex instruction comprehension.

The benchmark splits into two groups:

Task Aggregation (50 tasks)

Tests ability to handle linguistic diversity:

• Novel verbs: "retrieve," "transport," "relocate" instead of "pick and place"
• Multilingual expressions: instructions in multiple languages
• Diverse object references: "the beverage container" instead of "the bottle"
• Out-of-Distribution objects: objects not seen during training

Situated Reasoning (30 tasks)

Tests contextual inference:

• Contextual reasoning: "It's getting cold, please prepare something warm" → model infers: get a cup and pour hot water
• Command decomposition: long instructions split into actionable sub-goals
• Spatial reasoning: "put the item to the left of the blue object" → understand spatial relations

Results: The Numbers Speak

SimplerEnv (In-Domain Tasks)

InstructVLA-Expert outperforms SpatialVLA by +11% and OpenVLA by +87%.

SimplerEnv-Instruct (Instruction-Following Tasks)

+96% over fine-tuned OpenVLA, +29% over GPT-4o-assisted OpenVLA.

Notably, InstructVLA scores higher on Situated Reasoning (48.8%) than Task Aggregation (43.6%) — confirming that VLA-IT genuinely improves complex reasoning, not just surface-level pattern matching.

LIBERO Benchmark

InstructVLA achieves 95.8% average success rate across all four LIBERO task suites when fine-tuned in-domain — placing it among the top performers on this benchmark.

Multimodal Understanding

InstructVLA-Generalist preserves VLM capability nearly intact:

Minimal degradation on multimodal benchmarks confirms MoE LoRA + VLA-IT successfully preserves VLM knowledge.

Installation and Setup

Requirements

• Python 3.10+
• PyTorch 2.2.0 + CUDA 12.1
• GPU: at least 1× A100 40GB (inference), 8× A100 (training)
• RAM: 64GB+ recommended

Step 1: Install Environment

# Clone the repository
git clone https://github.com/InternRobotics/InstructVLA.git
cd InstructVLA

# Create conda environment
conda create -n instructvla python=3.10 -y
conda activate instructvla

# Install PyTorch with CUDA 12.1
pip install torch==2.2.0 torchvision==0.17.0 torchaudio==2.2.0 \
  --index-url https://download.pytorch.org/whl/cu121

# Core dependencies
pip install transformers==4.51.0 accelerate==1.3.0 peft==0.13.0
pip install numpy==1.26.4

# Flash Attention for faster inference
pip install flash-attn==2.5.5 --no-build-isolation

# Remaining dependencies
pip install -r pip_requirements.txt

Step 2: Install SimplerEnv for Evaluation

git clone https://github.com/YangS03/my_maniskill.git ManiSkill2_real2sim
cd ManiSkill2_real2sim && pip install -e .
cd ..

Step 3: Download Model Weights

# Create checkpoint directory
mkdir -p ckpt
# Download Eagle2-2B base model and InstructVLA LoRA adapters
# from Hugging Face — see full instructions on the GitHub repo

Step 4: Run Inference

import torch
from instructvla import InstructVLAModel

# Load model
model = InstructVLAModel.from_pretrained("ckpt/instructvla-generalist")
model.eval()

# Prepare input
observation = {
    "image": image_tensor,           # (1, 3, 448, 448) from main camera
    "wrist_image": wrist_tensor,     # (1, 3, 224, 224) from wrist camera
    "state": robot_state_tensor,     # (1, state_dim) joint angles + EE pose
    "instruction": "Place the red mug to the right of the blue bottle"
}

# Generate action
with torch.no_grad():
    actions = model.predict_action(observation)
    # actions: (1, action_horizon, action_dim)
    # action_dim = 7 (6 DoF EE pose + gripper)
    # action_horizon = 16 steps

Training from Scratch: 2-Stage Pipeline

Stage 1: Action Pre-Training

Stage 1 trains the action expert and latent action queries on manipulation data alone, without any VLM multimodal data. Goal: learn how to "do" before learning how to "understand."

python -m torch.distributed.run --nproc_per_node 8 \
  scripts/train_eagle_dual_v2_action_only_meta_query_v2.py \
  --vla.base_vlm "ckpt/Eagle2-2B" \
  --vla.global_batch_size 128 \
  --vla.per_device_batch_size 16 \
  --stage stage1 \
  --future_action_window_size 15

Stage 1 trains approximately 650M parameters (action expert + action queries).

Stage 2: VLA Instruction Tuning

Stage 2 adds language LoRA and the scalar gate head, co-training on three data sources:

• Multimodal VLM data (Bunny dataset, 2M samples)
• Manipulation data (RT-1, Bridge)
• VLA-IT dataset (650K samples)

python -m torch.distributed.run --nproc_per_node 8 \
  scripts/train_eagle_dual_v2_action_only_meta_query_v2.py \
  --vla.base_vlm "ckpt/Eagle2-2B" \
  --stage stage2 \
  --use_mm True \
  --fix_system1 True \
  --vla.global_batch_size 768 \
  --vla.learning_rate 5e-5 \
  --future_action_window_size 15

Critical note on the 1:7 ratio: The mix of 1 multimodal batch per 7 manipulation batches is not arbitrary. Ablation studies show that at 1:3 (too much language data), manipulation performance drops significantly. At 1:14 (too little), language capability degrades. The sweet spot is 1:7 — run your own ablation if your dataset distribution differs significantly.

Stage 2 tunes approximately 220M parameters (language LoRA + scalar gate).

Real-World Deployment: WidowX and Franka Research 3

The paper validates InstructVLA on two real platforms:

WidowX-250 (zero-shot, kitchen environment):

• Atomic instructions: +23.3% over OpenVLA
• Reasoning tasks (zero-shot): +46.7%

Franka Research 3 (few-shot, spatial and math tasks):

• Reasoning tasks (few-shot): +41.7% over OpenVLA
• Math-centric tasks: 2.5× over π0

A striking example: when given "The guest is arriving in 10 minutes, please help prepare," InstructVLA correctly interprets this as an implicit instruction to clear the table and set out beverages — while OpenVLA either freezes or executes the wrong action.

InstructVLA vs Other Approaches

InstructVLA isn't the first work on instruction following for robots, but its approach is meaningfully different:

InstructVLA takes the harder path (end-to-end, joint training) but achieves substantially better results on instruction-following tasks.

Key Takeaways for Practitioners

If you want to apply InstructVLA's ideas to your own project, here are the critical points:

1. Never full fine-tune on action data if you want to preserve language capability. Use PEFT (LoRA, MoE LoRA) and freeze most of the backbone.

2. Dataset quality beats dataset quantity. 650K GPT-4o-annotated samples with 4 rich annotation types outperform millions of simple instruction-action pairs.

3. The multimodal co-training ratio matters enormously. The 1:7 ratio was found through ablation — don't assume it generalizes to your dataset without testing.

4. Action queries are the architectural key. A learnable query layer between the VLM and action decoder reduces interference between language and action heads while preserving relevant information flow.

5. SimplerEnv-Instruct is the right benchmark to use. If you're building a VLA, test on SimplerEnv-Instruct — not just standard SimplerEnv — for a realistic picture of instruction generalization.

Conclusion

InstructVLA tackles a genuinely important problem: how can a robot simultaneously "do well" and "understand well" without sacrificing one for the other? Through VLA-IT, MoE LoRA, and 650K high-quality training samples, the paper demonstrates this is fully achievable.

+96% over OpenVLA on instruction following, 95.8% on LIBERO, and strong real-world performance on WidowX and Franka make InstructVLA one of the most compelling VLA frameworks for anyone building manipulation systems that need to understand natural language.

Resources:

• Paper: arXiv 2507.17520
• GitHub: InternRobotics/InstructVLA
• Dataset: ShuaiYang03/VLA_Instruction_Tuning
• Project page: yangs03.github.io/InstructVLA_Home

Related Posts

• OpenVLA Deep Dive: Architecture and Fine-tuning for Robot Manipulation
• SpatialVLA: VLA with Spatial Reasoning for Robot Manipulation
• A1 VLA with Truncated Flow Matching: Faster Inference for Robots