Fine-tune InternVLA-A1.5 with LeRobot

InternVLA-A1.5 is a substantial update from InternRobotics and Shanghai AI Lab for Vision-Language-Action (VLA) robot manipulation. InternVLA-A1 already tried to unify understanding, generation, and action in a Mixture-of-Transformers policy. A1.5 makes the design more deployment-friendly: it preserves the semantic strength of a native VLM, uses latent foresight tokens to learn dynamics from a pretrained video generation model, and removes the video branch at inference so the policy can still run in real time.

This guide is written for the practical question: if you have a LeRobot-format dataset and want to fine-tune InternVLA-A1.5 for your own manipulation task, what architecture should you understand, what environment do you need, how should the dataset be prepared, which training flags matter, and how do you sanity-check inference? The technical details below come from the arXiv paper 2607.04988, the project page, and the InternVLA-A-series GitHub repository.

If you have read our earlier InternVLA-A1 world-model guide, think of A1.5 as a very deliberate correction. Instead of forcing the policy to learn pixel-level future generation from scratch, A1.5 turns future prediction into a latent-querying problem. A small group of tokens learns to ask the shared multimodal context: "What future information is relevant for the next action chunk?" Those tokens are supervised by WAN2.2-5B during training. At deployment, the robot does not generate future videos.

Paper idea: Preserve semantics, learn dynamics, avoid deployment cost

VLA policies such as OpenVLA, π0/π0.5, GR00T, and many LeRobot-era policies share the same core bet: pretrained VLMs already contain rich visual and linguistic semantics, so we can attach action heads and teach robots to follow instructions. The problem is that manipulation is not just instruction understanding. The robot must also reason about dynamics: how an object will slide, where the gripper is heading, which stage of a long-horizon task is active, and how the current action changes the scene.

One response is to add a world model or video prediction branch. The A1.5 paper points out three common failure modes:

1. Semantic erosion: heavy action and video objectives can erode the original instruction-following ability of the pretrained VLM.
2. Objective interference: language modeling, future prediction, and continuous action regression have different loss scales and optimization behavior.
3. Expensive future learning: many policies learn future prediction from scratch, ignoring the dynamics priors already present in large video generators.

InternVLA-A1.5 addresses those issues with three design choices:

The key deployment point is simple: the video generation branch is used only during training and visualization. During real inference, you can use the optimized action-only backend and skip WAN entirely. Latent foresight becomes knowledge distilled into the policy, not an extra generator that must run at every control step.

Architecture: Qwen3.5 + unified expert + latent foresight

A1.5 has two main components.

The first is a Qwen3.5-2B VLM backbone. It consumes multi-view camera images, the language instruction, a control-mode token, and the robot proprioceptive state. The state is discretized with uniform binning and appended as tokens, which keeps the input close to the native VLM format instead of forcing everything through a separate robot encoder.

The second component is a unified expert with roughly 460M parameters. It follows the same broad hybrid design as Qwen3.5 text model blocks: 3 Gated DeltaNet linear-attention layers interleaved with 1 full-attention layer. The VLM backbone and unified expert exchange context only through the shared full-attention layers, while their Gated DeltaNet layers remain separate for modality-specific processing.

Inside the unified expert, two token groups matter:

• Foresight tokens: 50 learnable tokens that act as latent queries for future prediction. They attend to shared multimodal context, produce conditioning signals for frozen WAN2.2-5B, and receive video prediction loss during training.
• Action query tokens: query tokens for the action chunk. The action head uses flow matching to predict continuous action chunks. The default action chunk size is 50.

A simplified forward pass looks like this:

Multi-view images + instruction + state + mode
        |
        v
Qwen3.5-2B VLM backbone
        |
        | shared full attention
        v
Unified expert
        |-------------------------|
        v                         v
50 foresight tokens          action queries
        |                         |
WAN2.2-5B video loss          flow matching action loss
(training only)               continuous action chunk

For a LeRobot user, this explains why the public training script includes flags such as --policy.enable_vqa_loss=true, --dataset.use_fast_action_tokens=true, --policy.num_learnable_tokens=50, --policy.video_loss_weight=1, and --policy.action_loss_only=false. A1.5 is not merely fine-tuning an action decoder. It keeps language and VQA supervision alive so the policy does not forget instruction semantics.

Data recipe: 1.2M episodes, 861M frames, 3M multimodal samples

The paper and project page describe two large data streams.

The robot manipulation stream contains about 1.2M robot episodes and 861M frames from six sources. InternData-A1 provides the largest simulation foundation. Real-world sources such as AgiBotWorld, UMI, DROID, Galaxea, and RoboMind add embodiment diversity, viewpoints, and scenes. All sources are cast into the unified InternVLA-A1 action space, with morphology-specific slots padded into a shared layout so one action head can learn across robot types.

The multimodal co-training stream contributes around 3M samples in the InternVLA-M1 style: General QA, Box QA, Point QA, and Trajectory QA. This is not an optional decoration. It is how A1.5 protects semantic and spatial grounding while learning robot control. If you fine-tune only action loss on a few thousand demonstrations, the model can learn shortcuts: look at the current pose and replay common actions, without truly grounding the instruction.

For a small lab fine-tune, you do not need to reproduce the full pretraining corpus. But you should keep three habits:

1. Your dataset should contain clear instruction text, not only action arrays.
2. Camera views should be stable and close to the deployment setup.
3. Action normalization and statistics must match your abs or delta action mode.

If you are building the data pipeline from scratch, read the LeRobot ecosystem guide and our VLA deployment guide before running a large A1.5 job.

Installation

The main repository is InternRobotics/InternVLA-A-series. According to the official installation tutorial, the code has been tested with Python 3.11, CUDA 12.8, and PyTorch 2.10.0. Use Linux with NVIDIA GPUs. Real post-training is heavy; a consumer GPU may be enough for small open-loop tests, but the full 60K-step recipe is intended for stronger single-GPU or multi-GPU setups.

Basic environment setup:

conda create -y -n internvla_a1_5 python=3.11
conda activate internvla_a1_5
pip install --upgrade pip

conda install -c conda-forge ffmpeg svt-av1 -y

pip install torch==2.10.0 torchvision==0.25.0 \
  --index-url https://download.pytorch.org/whl/cu128

pip install transformers==5.2.0
pip install -e .

pip install flash-attn==2.8.3 flash-linear-attention==0.5.0 causal-conv1d==1.6.1 \
  --no-build-isolation

The repository also patches several Transformers modules for Qwen3.5 and robot-learning policies:

TRANSFORMERS_DIR=${CONDA_PREFIX}/lib/python3.11/site-packages/transformers/

cp -r src/lerobot/policies/pi0/transformers_replace/models ${TRANSFORMERS_DIR}
cp -r src/lerobot/policies/pi05/transformers_replace/models ${TRANSFORMERS_DIR}
cp -r src/lerobot/policies/internvla_a1_5/transformers_replace/models ${TRANSFORMERS_DIR}

Configure Hugging Face and LeRobot cache paths:

export HF_TOKEN=your_token
export HF_HOME=/data/huggingface
export HF_LEROBOT_HOME=${HF_HOME}/lerobot
ln -s ${HF_HOME}/lerobot data

If you train with video foresight supervision, download WAN2.2-TI2V-5B:

huggingface-cli download Wan-AI/Wan2.2-TI2V-5B \
  --local-dir ${HF_HOME}/hub/Wan2.2-TI2V-5B

If you only run action-only inference or evaluation, you can skip WAN. This is one of the practical benefits of A1.5: WAN is valuable during latent foresight learning, but the real robot does not have to pay the latency of video generation.

Prepare a LeRobot dataset

The official fine-tuning tutorial uses A2D Pick-Pen from InternData-A1 in LeRobot v2.1 format. The pipeline has four steps: download, extract, convert to LeRobot v3.0, and compute normalization statistics.

Download the dataset:

hf download \
  InternRobotics/InternData-A1 \
  real/genie1/Put_the_pen_from_the_table_into_the_pen_holder.tar.gz \
  --repo-type dataset \
  --local-dir data

Extract and organize it:

tar -xzf data/real/genie1/Put_the_pen_from_the_table_into_the_pen_holder.tar.gz -C data
mkdir -p data/v21
mv data/set_0 data/v21/a2d_pick_pen

Convert v2.1 to v3.0:

python src/lerobot/datasets/v30/convert_my_dataset_v21_to_v30.py \
  --old-repo-id v21/a2d_pick_pen \
  --new-repo-id v30/a2d_pick_pen

If you use delta actions, compute statistics for chunk size 50:

python util_scripts/compute_norm_stats_single.py \
  --action_mode delta \
  --chunk_size 50 \
  --repo_id v30/a2d_pick_pen

The stats file is typically written to:

${HF_HOME}/lerobot/stats/delta/v30/a2d_pick_pen/stats.json

Beginners often skip this step, but it is critical. If the stats are wrong, training loss may still decrease while unnormalized robot actions are badly scaled. A small mismatch in joint position, gripper value, or end-effector delta can make the robot overshoot, undershoot, or simply freeze.

Fine-tune InternVLA-A1.5

The repository provides a quick start on lerobot/pusht:

bash launch/internvla_a15_finetune.sh lerobot/pusht abs false

In this command:

• lerobot/pusht is the dataset repository id.
• abs selects absolute action mode.
• false tells the script to use the dataset-provided stats.json instead of an external stats path.

For your converted custom dataset, copy the launch script and adjust the important variables:

POLICY="internvla_a1_5"
PRETRAINED_PATH="InternRobotics/InternVLA-A1.5-base"
DATASET_REPO_ID="v30/a2d_pick_pen"
ACTION_TYPE="delta"
USE_EXTERNAL_STATS=true

Important policy flags:

--policy.type=internvla_a1_5
--policy.pretrained_path=InternRobotics/InternVLA-A1.5-base
--policy.dtype=bfloat16
--policy.optimizer_lr=5e-5
--policy.scheduler_warmup_steps=2000
--policy.scheduler_decay_steps=60000
--policy.scheduler_decay_lr=5e-6
--policy.freeze_vision_encoder=false
--policy.train_expert_only=false
--policy.enable_vqa_loss=true
--policy.tokenize_state=true
--policy.video_loss_only=false
--policy.video_loss_weight=1
--policy.action_loss_only=false
--policy.freeze_learnable_tokens=true
--policy.num_learnable_tokens=50

Important dataset flags:

--dataset.type=internvla_a1_5
--dataset.repo_id=v30/a2d_pick_pen
--dataset.action_mode=delta
--dataset.use_external_stats=true
--dataset.external_stats_path=${HF_HOME}/lerobot/stats/delta/v30/a2d_pick_pen/stats.json
--dataset.tokenize_state=true
--dataset.use_fast_action_tokens=true

Training flags:

--batch_size=8
--steps=60000
--save_freq=20000
--log_freq=200

The paper uses post-training batch size 128, cosine learning-rate decay from 5e-5 to 5e-6 over 60K steps, 2K warmup steps, weight decay 0.01, gradient clipping 1.0, and bfloat16 precision. The public script defaults to batch size 8 to make smaller runs easier. If you have multiple GPUs, scale PROC_PER_NODE, the global batch size, and gradient accumulation according to VRAM.

A practical lab recipe:

For a static pick-and-place task, A1.5 may be heavier than ACT or Diffusion Policy. But if your task involves instruction bindings, compositional objects, multiple stages, or scene changes caused by previous actions, latent foresight becomes much more relevant.

Inference and evaluation

Before touching a real robot, run open-loop evaluation:

python tests/openloop_internvla_a1_5.py \
  --ckpt-path outputs/internvla_a1_5/your_checkpoint \
  --dataset-root data/v30/a2d_pick_pen \
  --out-dir outputs/openloop/a2d_pick_pen \
  --num-episodes 2 \
  --max-samples-per-episode 8

This script loads a checkpoint, reads a LeRobot demo dataset, predicts chunked actions, unnormalizes the action layout, plots trajectories, and can save future-frame visualizations when --visualize-future is enabled. For a first pass, inspect three things:

1. Are predicted actions on the same scale as ground truth actions?
2. Does gripper open/close timing match the task phase?
3. Does the action chunk jump sharply between adjacent timesteps?

For real-robot deployment, the repository recommends optimized action-only inference:

config.inference_backend, config.action_loss_only = "optimized", True

This skips WAN and predicts actions only. According to the paper, real-world experiments were run on a single RTX 5090; with static-graph execution, SDPA, and flash linear attention, one inference step takes roughly 0.1 seconds. Your number will depend on camera capture, preprocessing, GPU, robot middleware, and control frequency.

For simulation benchmarks, the repository provides separate workflows for LIBERO, LIBERO-Plus, RoboTwin, DOMINO, and SimplerEnv. LIBERO uses a policy server plus simulator client flow. Important variables include CKPT_PATH, LIBERO_HOME, GPU_IDS, NUM_TRIALS_PER_TASK, REPLAN_STEPS, and INFERENCE_BACKEND.

Results: A1.5 is strongest at compositional generalization

Headline simulation results from the paper:

On RoboTwin, A1.5 barely drops from clean to randomized evaluation: 93.3 versus 93.0. On DOMINO, a dynamic manipulation benchmark with moving objects, it reaches 27.7% zero-shot success rate and 29.3% after fine-tuning. This is where latent foresight is more meaningful than in purely static manipulation.

The ablation table is also important:

In short, removing video supervision or removing foresight tokens hurts, especially on LIBERO-Plus and DOMINO. This supports the central claim of the paper: foresight tokens are not decorative; they are the interface that distills dynamics priors from a video model into an action policy.

In the real world, the paper evaluates Sort Tubes, Insert Tubes, Move Tubes, and a long-horizon MOF chemistry task. InternVLA-A1.5 leads on Insert Tubes, Move Tubes, and MOF. Sort Tubes is slightly behind a strong baseline overall, but A1.5 remains strong on held-out instruction bindings. On MOF, A1.5 reaches 76.4%, while π0.5 reaches 29.3% and Motus does not complete any trial. For a multi-stage procedure, subtask prediction and dynamics priors help the policy track task progress rather than simply execute a local pick action.

Fine-tuning checklist

Before launching a 60K-step job, verify:

• Your dataset has the correct LeRobot v3.0 layout: data/, meta/, and videos/.
• The action_mode in stats and training script matches: abs or delta.
• chunk_size=50 is consistent across transforms, stats, and policy config.
• 224x224 image resizing does not crop out the end-effector or main object.
• Instruction text is consistent but not impoverished; if every episode has the same sentence, you should not expect compositional generalization.
• HF_HOME, HF_LEROBOT_HOME, WANDB_TOKEN, CONDA_ROOT, CUDA path, and output directory are correct in the launch script.
• For inference only, use the optimized action-only path to avoid unnecessary WAN loading.

Compared with related work such as VLA-JEPA latent world models or Weaver for π0.5, the trend is clear: 2026-era VLA policies are moving beyond direct image-language-to-action mapping. They increasingly learn representations of the future. InternVLA-A1.5 stands out because that future representation is learned from a large video generator, while deployment remains action-only.

Conclusion

InternVLA-A1.5 is worth studying not only because its benchmarks are strong, but because it provides a clean manipulation VLA pattern: preserve a native VLM for semantics, add a lightweight expert for action, use latent foresight to distill dynamics from a video model, then remove the video branch at inference. With LeRobot, the public repository now gives a concrete path: install the environment, convert v2.1 data to v3.0, compute stats, run internvla_a15_finetune.sh, validate with open-loop evaluation, and deploy through the optimized action-only backend.

For a lab or product stack, start small: 200-500 episodes, a clear instruction family, fixed cameras, and clean action statistics. Once the pipeline is reliable, expand into compositional instructions, more object bindings, and long-horizon tasks. That is where A1.5 is most likely to justify its complexity.

Related Posts

• InternVLA-A1: VLA + World Model
• LeRobot ecosystem guide
• PEFT/LoRA Fine-tuning & VLA Deployment