Fine-Tuning EVO1 on SO-101 with LeRobot v0.6.0

Most Vision-Language-Action models face an uncomfortable tradeoff: the more capable they are, the heavier they become. π₀ needs 3.5B parameters. OpenVLA requires 7B. Both demand A100 or H100 GPUs just to run inference at 4–5 Hz — reasonable for a well-funded lab, but impractical for edge robotics or budget setups.

EVO1, from MINT Lab at Shanghai Jiao Tong University, challenges this assumption. At just 0.77B parameters, it outperforms SmolVLA (2.25B), OpenVLA-OFT (7B), and even π₀ (3.5B) across multiple real-robot manipulation benchmarks. More importantly: it runs at 16.4 Hz using only 2.3 GB of VRAM on a consumer GPU — deployable on an SO-101 robotic arm with no cloud dependency.

This is a complete tutorial for fine-tuning EVO1 on the SO-101 robot arm using LeRobot v0.6.0, which ships EVO1 as a first-class policy type.

What Is EVO1?

EVO1 was introduced in "Evo-1: Lightweight Vision-Language-Action Model with Preserved Semantic Alignment" by MINT Lab (Shanghai Jiao Tong University), published on arXiv in November 2025 and accepted at CVPR 2026 (Efficient CVPR Badge).

The name "Evo" captures the core philosophy: evolution over scale. Instead of following the trend of pushing model size larger, the team searched for architectural improvements that achieve better results with fewer parameters. The central technical contribution is a two-stage training paradigm that preserves the semantic alignment of the VLM backbone while teaching it to control a robot.

Why does "semantic alignment" matter? When you fine-tune a VLM (InternVL3, LLaVA) end-to-end for robot control from scratch, the gradients from the action loss gradually distort the VLM's internal representations — it learns to associate tokens with motor commands but loses the ability to generalize to unseen objects or tasks. EVO1's two-stage design specifically addresses this degradation.

Architecture

EVO1 overall architecture — source: MINT-SJTU/Evo-1

EVO1 has three main components connected in sequence:

1. InternVL3-1B Backbone

The backbone is InternVL3-1B, combining:

• InternViT-300M: vision encoder that processes 448×448 RGB images
• Qwen2.5-0.5B (first 14 layers only): language decoder that understands task instructions

Why InternVL3? It was pre-trained under a native multimodal paradigm — vision and language trained jointly from scratch rather than stitched together post-hoc. This creates tighter cross-modal alignment. When fine-tuned for robot control, this alignment is what enables generalization to novel objects and scenes. The native HuggingFace checkpoint to use is OpenGVLab/InternVL3-1B-hf (note the -hf suffix — required by LeRobot's transformers implementation).

2. Integration Module

This is EVO1's most distinctive design choice. Rather than feeding the VLM's final-layer output into the action head, the Integration Module taps into mid-layer features of InternVL3 (specifically layer 14) and concatenates them directly with the robot's proprioceptive state (joint angles + gripper state).

Mid-layer features contain richer spatial and semantic information than the final layer, which tends to collapse into a narrow distribution suited for language generation rather than robot control. By hooking into this intermediate representation, the action head receives a more informative embedding without needing to unroll the entire VLM.

3. Cross-Modulated Diffusion Transformer (Action Head)

The action head is a Diffusion Transformer using flow-matching:

• Query: noise-injected action tokens (random initialization)
• Key-Value: multimodal embeddings from the Integration Module combined with the state vector
• Cross-attention iteratively denoises the action chunk, conditioned on the visual and language context
• Output: a chunk of 50 continuous joint position commands

The key efficiency gain over DDPM-style diffusion: EVO1 only needs 32 denoising steps to sample a complete action chunk, which is why inference reaches 16.4 Hz on consumer hardware.

Compare this with SmolVLA's architecture — similar flow-matching DiT, but without the Integration Module's mid-layer tap and without the two-stage training that preserves semantic alignment.

Two-Stage Training: The Core Innovation

If you fine-tune the full VLM and action head simultaneously from the beginning, the action loss gradients distort the VLM's attention patterns. The model starts attending to regions that correlate with motor commands rather than the semantic content of the task instruction. The result: poor generalization to new objects, backgrounds, or positions.

Stage 1 — Action Head Warmup (5,000 steps):

• Freeze the entire InternVL3 backbone
• Train only the Integration Module and DiT action head
• The VLM provides stable, high-quality features; the action head learns to decode them into motor commands
• Result: the action head converges on a reasonable warm start with stable gradients

Stage 2 — Joint Fine-Tuning (up to 80,000 steps for LIBERO; ~50,000 for SO-101):

• Unfreeze the VLM; train everything end-to-end
• With the action head already warmed up, VLM gradients are gentler and more structured
• The VLM adapts minimally and retains semantic alignment
• Result: the model generalizes to unseen objects because the VLM still "understands" language

Ablation studies in the paper confirm this: VLM attention maps after Stage 2 continue to focus correctly on the object mentioned in the task instruction, whereas single-stage baselines suffer attention collapse.

Benchmark Results

¹ SmolVLA in the paper benchmark uses the full PaliGemma-backbone variant (2.25B). SmolVLA-Base is 450M but doesn't include a VLM.

LeRobot LIBERO, Stage 2 checkpoint at step 70,000 (single seed, 2× H100):

The most remarkable result: on real robot (xArm6), EVO1 at 0.77B outperforms π₀ at 3.5B — a model nearly 5× larger — with 78% vs 73% success rate. This validates the semantic alignment hypothesis: a smaller model that generalizes better is more useful than a larger one that overfits to training motions.

Installation

Hardware Requirements

• Python 3.10+ and CUDA 11.8+
• Stage 1: GPU with ≥ 12 GB VRAM (RTX 3060 12GB is sufficient)
• Stage 2: GPU with ≥ 24 GB VRAM (RTX 4090 or A100 recommended; RTX 3090 24GB works with smaller batch)
• Inference: GPU with ≥ 4 GB VRAM (the paper ran SO-101 inference on an RTX 4060)

Install LeRobot with EVO1 Extra

# Clone LeRobot
git clone https://github.com/huggingface/lerobot.git
cd lerobot

# Install with evo1 extra (includes InternVL3 dependencies)
pip install -e ".[training,evo1]"

# Optional but recommended: Flash Attention for ~20% faster inference
# Only install if compatible with your Python + PyTorch + CUDA stack
pip install flash-attn --no-build-isolation

Critical note on VLM checkpoint: policy.vlm_model_name must point to OpenGVLab/InternVL3-1B-hf — note the -hf suffix. LeRobot uses the native HuggingFace transformers implementation of InternVL3, not the original MINT-SJTU format. Omitting -hf causes a tokenizer load error on the first run.

The first run will automatically download the InternVL3-1B-hf checkpoint (~2 GB) to your HuggingFace cache. Subsequent runs reuse the cache.

Preparing SO-101 Dataset

EVO1 expects standard LeRobot dataset format. For SO-101 (6-DOF arm + gripper = 7D action space):

observation.images.top      # Top-down or front-facing camera (required)
observation.images.wrist    # Wrist-mounted camera (optional, improves success ~10-15%)
observation.state           # Joint angles + gripper state (7D)
action                      # Target joint positions (7D)
task                        # Language instruction ("Pick up the red cube and place it in the bowl")

Collecting Teleoperation Data

lerobot-record \
  --robot.type=so101 \
  --dataset.repo_id=your_hf_username/so101_pick_place \
  --dataset.task="Pick up the red cube and place it in the bowl" \
  --dataset.fps=30 \
  --dataset.num_episodes=100 \
  --cameras.top=0 \
  --cameras.wrist=1

How many demonstrations? 50–100 demos for simple single-object tasks. 100–200 for multi-object or more complex manipulation. The original SO-101 experiments used 100 demos per task. EVO1's VLM backbone enables better generalization from smaller datasets compared to policy-only approaches — you'll notice this especially when testing with unseen object colors.

Verify the Dataset

from lerobot.common.datasets.lerobot_dataset import LeRobotDataset

dataset = LeRobotDataset("your_hf_username/so101_pick_place")
print(f"Episodes: {dataset.num_episodes}")
print(f"Total frames: {len(dataset)}")
print(f"Features: {list(dataset.features.keys())}")
# Expected output:
# Episodes: 100
# Total frames: ~9000 (90 frames/episode × 100)
# Features: ['observation.images.top', 'observation.images.wrist', 'observation.state', 'action', 'task', ...]

Stage 1: Action Head Warmup

Stage 1 freezes InternVL3 and trains only the Integration Module + DiT action head. Run for 5,000 steps — enough for the action head to learn coarse motion patterns without overfitting.

lerobot-train \
  --dataset.repo_id=your_hf_username/so101_pick_place \
  --policy.type=evo1 \
  --policy.training_stage=stage1 \
  --policy.vlm_model_name=OpenGVLab/InternVL3-1B-hf \
  --policy.device=cuda \
  --policy.chunk_size=50 \
  --policy.n_action_steps=50 \
  --policy.max_state_dim=24 \
  --policy.max_action_dim=24 \
  --policy.dropout=0.2 \
  --policy.optimizer_lr=1e-5 \
  --policy.optimizer_weight_decay=1e-3 \
  --policy.optimizer_grad_clip_norm=1.0 \
  --policy.use_amp=true \
  --policy.enable_gradient_checkpointing=true \
  --policy.gradient_checkpointing_use_reentrant=false \
  --batch_size=8 \
  --steps=5000 \
  --save_freq=2500 \
  --output_dir=./outputs/evo1_so101_stage1

Key parameters explained:

Expected Stage 1 behavior: Loss should drop from ~0.08–0.12 to ~0.02–0.04 within 5,000 steps. If loss isn't decreasing, check that your dataset has the task field populated and that the learning rate isn't too high.

The Stage 1 checkpoint is saved at ./outputs/evo1_so101_stage1/checkpoints/005000/pretrained_model.

Stage 2: Joint VLM Fine-Tuning

Stage 2 loads the Stage 1 checkpoint and unfreezes the VLM for full end-to-end training. This is the step that determines final policy quality.

lerobot-train \
  --dataset.repo_id=your_hf_username/so101_pick_place \
  --policy.path=./outputs/evo1_so101_stage1/checkpoints/005000/pretrained_model \
  --policy.training_stage=stage2 \
  --policy.vlm_model_name=OpenGVLab/InternVL3-1B-hf \
  --policy.device=cuda \
  --policy.chunk_size=50 \
  --policy.n_action_steps=50 \
  --policy.max_state_dim=24 \
  --policy.max_action_dim=24 \
  --policy.dropout=0.2 \
  --policy.optimizer_lr=1e-5 \
  --policy.optimizer_weight_decay=1e-3 \
  --policy.optimizer_grad_clip_norm=1.0 \
  --policy.vlm_dtype=bfloat16 \
  --policy.use_amp=true \
  --policy.use_flash_attn=true \
  --policy.enable_gradient_checkpointing=true \
  --policy.gradient_checkpointing_use_reentrant=false \
  --batch_size=4 \
  --steps=50000 \
  --save_freq=5000 \
  --output_dir=./outputs/evo1_so101_stage2

On policy.training_stage=stage2: This flag automatically applies Stage 2 defaults — unfreeze VLM, enable joint fine-tuning. You don't need to pass --policy.finetune_vlm=true separately. Add --policy.apply_training_stage_defaults=false only if you need manual control over every fine-tuning flag.

How many steps for SO-101? The original SO-101 paper experiments used 50,000 steps (batch 16, 1× A100) and achieved 78% real-world success. The 80,000-step recipe is for LIBERO with a larger dataset. With 100 episodes (~9,000 frames), you may see overfitting after 30–40k steps. Set --save_freq=5000 and pick the best checkpoint by validation loss.

On policy.vlm_dtype=bfloat16: BF16 is more gradient-stable than FP16 when fine-tuning language models. If your GPU predates RTX 30xx (no native BF16), use float16, but expect slightly less stable training.

The same two-stage workflow compares well against MolmoAct2's SO-101 guide: MolmoAct2 uses single-stage LoRA fine-tuning (simpler, faster convergence), while EVO1's two-stage approach offers more control over when the VLM gets updated. Trade-off: EVO1 takes longer to train but has simpler inference (no LoRA adapter management).

Inference on the Real Robot

After training, deploy with lerobot-rollout:

lerobot-rollout \
  --policy.path=./outputs/evo1_so101_stage2/checkpoints/best \
  --policy.vlm_model_name=OpenGVLab/InternVL3-1B-hf \
  --policy.device=cuda \
  --policy.use_amp=true \
  --policy.vlm_dtype=bfloat16 \
  --policy.use_flash_attn=true \
  --policy.chunk_size=50 \
  --policy.n_action_steps=14 \
  --policy.max_state_dim=24 \
  --policy.max_action_dim=24 \
  --inference.type=rtc \
  --robot.type=so101 \
  --cameras.top=0 \
  --cameras.wrist=1

--inference.type=rtc activates Real-Time Chunking (RTC), a LeRobot v0.6.0 feature that computes the next action chunk in parallel while the robot executes the current one. This reduces effective latency by ~30% compared to sequential inference.

Why n_action_steps=14 for inference instead of 50? This is an intentional trade-off: larger chunks (50) produce smoother motion but respond slowly to disturbances; smaller chunks (14) replan more frequently, making the robot more responsive when objects shift or the arm drifts. For pick-and-place tasks on SO-101, 14–25 steps is the sweet spot. Experiment to find what works for your task.

Using the pretrained SO-101 checkpoint: Instead of fine-tuning from scratch, you can start from the MINT-SJTU checkpoint trained specifically on SO-100/SO-101:

lerobot-rollout \
  --policy.path=MINT-SJTU/Evo1_SO100 \
  --policy.vlm_model_name=OpenGVLab/InternVL3-1B-hf \
  --inference.type=rtc \
  --robot.type=so101 \
  ...

This is useful for zero-shot evaluation before committing to fine-tuning on your specific task.

Real-World Performance and What to Expect

Based on the original SO-101 experiments (100 demos per task, RTX 4060 inference):

• Pick and place (single object, fixed position): ~85–90% success rate
• Pick and place (multiple objects, natural language instruction): ~75–80%
• Generalization to unseen object colors: ~60–70% — this is where EVO1 outperforms flow-matching policies without preserved VLM alignment

For deeper context on flow-matching in VLA models, read A1 VLA: 72% Lower Latency with Truncated Flow Matching — a parallel approach that reduces denoising steps even further using inter-layer truncation.

Optimization Tips for Consumer GPUs

1. Gradient checkpointing is mandatory for Stage 2:

--policy.enable_gradient_checkpointing=true
--policy.gradient_checkpointing_use_reentrant=false

Reduces Stage 2 VRAM from ~20 GB to ~12–14 GB. ~20% slower training speed, but necessary for GPUs under 24 GB.

2. Batch size and gradient accumulation: Stage 2, RTX 4090 (24 GB): --batch_size=4 works fine. For RTX 3090 (24 GB):

--batch_size=2 --gradient_accumulation_steps=4

Maintains effective global batch size of 8.

3. Flash Attention saves another 15–20% VRAM:

pip install flash-attn --no-build-isolation

EVO1 auto-detects Flash Attention when installed (default use_flash_attn=true). No extra flag needed.

4. Multi-GPU with Accelerate: With 2 GPUs, Stage 2 trains ~1.8× faster:

accelerate launch --num_processes=2 -m lerobot.scripts.lerobot_train \
  --policy.type=evo1 \
  --policy.training_stage=stage2 \
  # ... (same parameters as above)

5. No large-scale robot pretraining data required: EVO1 starts from OpenGVLab/InternVL3-1B-hf pretrained on internet data. Unlike π₀ (which relies on dPlan with millions of robot demonstrations), EVO1 doesn't require a proprietary robot pretraining dataset. This makes it practical for small labs, academic research, and individual roboticists.

Summary

EVO1 fills a real gap in the VLA ecosystem: small enough to run real-time on a consumer GPU, capable enough to outperform much larger models on real manipulation tasks. Its two-stage training paradigm is a useful design pattern — freezing the VLM backbone first before joint fine-tuning isn't slower by accident, it's a deliberate choice that prevents semantic alignment degradation.

For SO-101 with LeRobot v0.6.0, EVO1 is one of the most practical options available for a lab wanting to try VLA without investing in GPU clusters. Stage 1 runs on an RTX 3060 12GB; Stage 2 needs 24GB but stays within consumer-grade hardware.

Resources:

• Paper (CVPR 2026): arXiv 2511.04555
• GitHub: MINT-SJTU/Evo-1
• SO100/SO101 checkpoint: MINT-SJTU/Evo1_SO100
• LIBERO checkpoint (community): zuoxingdong/evo1_libero
• LeRobot docs: huggingface.co/docs/lerobot/en/evo1
• Jetson Orin Docker: MINT-SJTU/Evo-1_JetsonOrin

Related Posts

• Fine-Tuning MolmoAct2 on SO-101 with LeRobot v0.6 — Same workflow for MolmoAct2, which uses single-stage LoRA and outperforms π₀.5 in out-of-distribution settings
• SmolVLA: Training a 450M VLA on a Consumer GPU — Architectural comparison: SmolVLA has a simpler single-stage setup; EVO1's two-stage training delivers better generalization
• A1 VLA: 72% Lower Latency with Truncated Flow Matching — Deep dive into flow-matching variants for VLA inference optimization