Fine-tune MolmoAct2 on SO-101 with LeRobot v0.6

In May 2026, the Allen Institute for AI (Ai2) released MolmoAct2 — a fully open-source Vision-Language-Action model for robot manipulation. The headline result: on real-world out-of-distribution manipulation with a Franka Emika robot, MolmoAct2 reached 87.1% task success, while Physical Intelligence's π0.5 achieved 45.2% — nearly double the performance gap. On LIBERO (the standard simulation benchmark), all three top models cluster together: MolmoAct2 at 97.2%, GR00T N1.7 at 97.0%, π0.5 at 96.9%. But real-world generalization is where MolmoAct2 pulls ahead.

This tutorial walks through fine-tuning MolmoAct2 on the SO-101 robot arm using LoRA and LeRobot v0.6 — a pipeline that fits on a single 24GB GPU (RTX 3090/4090/A10G) with no cluster required. We'll cover the paper's key ideas, architecture, environment setup, data collection, training, and inference.

If you're new to LeRobot, read LeRobot Framework: VLA Basics from Scratch first.

What Is MolmoAct2 and Why Does It Matter?

MolmoAct2 is the successor to MolmoAct (Ai2, 2024). Full paper: MolmoAct2: Action Reasoning Models for Real-world Deployment, arXiv May 2026, 29 authors from Ai2 and the University of Washington.

The core distinction from previous VLAs: MolmoAct2 doesn't just optimize for simulation benchmarks like LIBERO. It targets real-world robustness — generalization when lighting, object positions, camera angles, and objects themselves differ from training data. The paper calls this "out-of-distribution" (OOD) evaluation, and it's where most manipulation policies fall apart.

Benchmark Comparison

The simulation benchmarks are close. Real-world OOD is the real test — that's where MolmoAct2's embodied reasoning backbone makes the difference.

Crucially, Ai2 released everything: checkpoints, training code, datasets, and a pre-trained checkpoint specifically for SO-100/101. No black box, no API-only access.

Architecture: Four Interlocking Components

MolmoAct2 is built from four components, each solving a distinct problem.

1. Molmo2-ER — The Embodied Reasoning Backbone

The backbone isn't the original 7B Molmo — instead it's Molmo2-ER, built on Qwen3-4B. Smaller parameter count, but trained with a specialized two-stage recipe on 3.3M samples:

• Stage 1 (20K steps): Embodied specialization — spatial reasoning, object pointing, grounding
• Stage 2 (1.5K steps): Interleave embodied + general data to prevent catastrophic forgetting

Result: 63.8/100 on 13 embodied-reasoning benchmarks — ahead of GPT-5, Gemini Robotics ER-1.5, and Qwen3-VL-8B. A backbone that understands 3D space more precisely translates directly into more accurate manipulation.

2. OpenFAST — Open Action Tokenizer

OpenFAST is a fully open action tokenizer. It maps 1-second continuous action trajectories into a 2,048-token vocabulary, trained on 1 million sequences from 5 embodiments (SO-100/101, YAM bimanual, DROID Franka, BC-Z, BridgeData V2).

Mechanism: frequency-domain transforms + byte-pair encoding to compress trajectories into token sequences. This enables autoregressive prediction like a language model — simple, scales well, no complex action head required.

3. Two-Phase Training: Discrete → Continuous

MolmoAct2 trains in two phases:

Phase 1: Pre-training (200K steps)
  → Autoregressive VLA predicts OpenFAST tokens (discrete)
  → Learns the "language" of actions

Phase 2: Post-training (100K steps)
  → Attaches DiT-style flow-matching action expert
  → Bridge via KV-cache conditioning (zero-copy)
  → Output: continuous action trajectory

The key innovation: KV-cache conditioning bridge — instead of replacing the backbone with an action head, the DiT expert reads the VLM's KV-cache directly, zero tensor copy overhead.

4. MolmoAct2-Think — 37x Faster Reasoning

The -Think variant selectively re-predicts depth tokens only for scene regions that changed (10×10 grid, 128 depth codes, cosine similarity threshold 0.996). When the robot executes a simple motion, -Think skips reasoning — only re-reasons when necessary. Result: 37x faster than the predecessor, achieving 98.1% on LIBERO with -Think.

Dataset MolmoAct2-SO100_101 — No Need to Start from Scratch

Ai2 released three large datasets:

The SO100_101 dataset went through 4-stage quality filtering: episodes without language removed, action range validation, length filtering, deduplication. This is your starting point if you want to fine-tune for a new task on SO-101 without collecting 38k demos yourself.

Pre-trained checkpoint: allenai/MolmoAct2-SO100_101 — use it zero-shot on SO-101 before deciding whether to fine-tune.

Hardware: SO-101 and GPU Requirements

SO-101 Robot Arm

SO-101 is the second-generation of SO-100, developed by TheRobotStudio in collaboration with HuggingFace. Repo: TheRobotStudio/SO-ARM100.

Standard setup: one follower arm (robot that executes) + one leader arm (human-operated for teleoperation). The follower records the leader's trajectory as demonstration data.

Cost: Self-build (3D print + hardware) approximately $100–130 per arm; two-arm leader+follower kit around $230. Third-party kits range $199–500.

Improvements over SO-100:

• Better cable routing — eliminates joint-3 disconnection risk
• Easier assembly — no gear removal during build
• Optimized gear ratios for leader arm

GPU Requirements

RTX 3090 (24GB) or RTX 4090 (24GB) works for LoRA. A10G (24GB) on cloud also works.

Environment Setup

Requirements

• Python 3.10 or 3.11
• CUDA 12.1+ (check with nvidia-smi)
• uv package manager (significantly faster than pip)

Install LeRobot v0.6 with MolmoAct2 extras

# Clone and install LeRobot
git clone https://github.com/huggingface/lerobot.git
cd lerobot

# Install with molmoact2 extras (flash-attention, accelerate, peft)
uv sync --locked --extra molmoact2

# Activate venv
source .venv/bin/activate

# Verify
python -c "from lerobot.policies.molmoact2 import MolmoAct2Policy; print('OK')"

The --extra molmoact2 flag pulls flash-attn, peft, and accelerate — all three required for LoRA fine-tuning.

Find and Configure SO-101 Port

Plug in the SO-101 follower via USB, then:

python -m lerobot.scripts.find_motors_bus_port

This returns the port (usually /dev/ttyACM0 on Linux). Save it for calibration.

Calibrate SO-101

python -m lerobot.scripts.control_robot \
  --robot.type=so101_follower \
  --robot.port=/dev/ttyACM0 \
  --control.type=calibrate

The script guides you through moving each joint to its extreme positions to record offsets. Calibration saves to ~/.cache/lerobot/calibration/so101_follower.json.

Data Collection via Teleoperation

Teleoperation: you control the leader arm, the follower records the trajectory as a demonstration:

python -m lerobot.scripts.control_robot \
  --robot.type=so101_follower \
  --robot.port=/dev/ttyACM0 \
  --teleop.type=so101_leader \
  --teleop.port=/dev/ttyACM1 \
  --control.type=record \
  --dataset.repo_id=${HF_USER}/my_so101_task \
  --dataset.task="pick up the red cube and place it in the blue box" \
  --dataset.num_episodes=50 \
  --dataset.fps=30

Each episode records: observations (camera RGB + proprioceptive state) + actions (joint positions). Dataset uploads automatically to HuggingFace Hub.

Tips for good demonstration data:

• 50–100 episodes is sufficient for a simple pick-and-place task
• Vary object positions, lighting, and camera angle slightly between episodes
• Move slowly and smoothly — avoid sudden jerks
• Delete failed episodes immediately — don't leave them in the dataset

If you want to use existing data: MolmoAct2-SO100_101 already has 38k episodes on HuggingFace.

LoRA Fine-tuning

LoRA vs Full Fine-tune

LoRA (Low-Rank Adaptation) doesn't train all model weights — it adds small low-rank matrix pairs to attention and MLP layers. Benefits:

• Only trains ~1–5% of total parameters → significantly less VRAM
• Prevents catastrophic forgetting (backbone retains general representations)
• Often outperforms FFT on small datasets (<200 episodes)

MolmoAct2 supports three VLM training modes (--policy.train_mode_vlm):

• lora — trains LoRA adapters on VLM, action expert fully trainable (recommended for small datasets)
• fft — full fine-tuning of VLM (requires 80GB+ VRAM or multi-GPU)
• freeze — freezes VLM entirely, only trains action expert

For <200 episodes on a single 24GB GPU: use lora.

LoRA Fine-tune Command (Single GPU)

accelerate launch \
  --num_processes=1 \
  --mixed_precision=bf16 \
  -m lerobot.scripts.lerobot_train \
  --policy.type=molmoact2 \
  --policy.checkpoint_path=allenai/MolmoAct2 \
  --policy.train_mode_vlm=lora \
  --policy.dtype=bfloat16 \
  --policy.gradient_checkpointing=true \
  --policy.action_mode=continuous \
  --policy.num_flow_timesteps=8 \
  --dataset.repo_id=${HF_USER}/my_so101_task \
  --batch_size=4 \
  --steps=10000 \
  --output_dir=outputs/molmoact2_so101_lora \
  --save_steps=1000

Key flag breakdown:

Default learning rates:

• VLM LoRA LR: 5e-5
• Action expert LR: 5e-5 (override with --policy.optimizer_action_expert_lr)

Monitor Training

# Loss curve
tensorboard --logdir=outputs/molmoact2_so101_lora

# VRAM usage
watch -n2 nvidia-smi --query-gpu=memory.used,memory.free --format=csv

Loss should decrease steadily in the first 3,000–5,000 steps then plateau. If loss oscillates strongly, reduce LR to 2e-5.

Upload Checkpoint

python -m lerobot.scripts.push_policy \
  --policy.path=outputs/molmoact2_so101_lora \
  --hub.repo_id=${HF_USER}/molmoact2-so101-task

Zero-shot with Pre-trained Checkpoint

If your task is similar to what's in the SO100_101 dataset (simple pick-and-place), try zero-shot before fine-tuning:

lerobot-rollout \
  --policy.path=lerobot/MolmoAct2-SO100_101-LeRobot \
  --robot.type=so100_follower \
  --robot.port=/dev/ttyACM0 \
  --task="pick up the red cube" \
  --duration=30

Zero-shot works well when: camera setup matches training data, task language is clear, objects are familiar. If success rate is below 60%, collect demos and fine-tune with LoRA.

Inference and Evaluation

Rollout with Fine-tuned Policy

lerobot-rollout \
  --policy.path=${HF_USER}/molmoact2-so101-task \
  --robot.type=so101_follower \
  --robot.port=/dev/ttyACM0 \
  --task="pick up the red cube and place it in the blue box" \
  --num_episodes=20 \
  --output_dir=eval_results/

--num_episodes=20 runs 20 evaluation episodes and reports success rate — enough for a preliminary estimate.

Automated Evaluation with lerobot-eval

LeRobot v0.6 added the lerobot-eval CLI with 6 simulation benchmarks. LIBERO-plus is the most relevant for manipulation:

lerobot-eval \
  --policy.path=${HF_USER}/molmoact2-so101-task \
  --env.type=libero \
  --eval.num_episodes=20 \
  --eval.task_suite=libero_spatial

Tuning Denoising Steps

num_flow_timesteps=8 is the default, well-balanced for most tasks. Try:

• 4 steps — 2x faster, acceptable for simple tasks
• 16 steps — 2x slower, sometimes improves complex multi-step tasks

Common Troubleshooting

CUDA OOM during training:

# Reduce batch size
--batch_size=2
# Or add gradient accumulation
--gradient_accumulation_steps=4

Loss not decreasing after 2000 steps:

• Check dataset format — verify action dimension matches SO-101 (6 joints + gripper = 7)
• Try increasing LR to 1e-4
• Check normalization — data must be normalized to SO-101's action range

Robot moves jerkily during inference:

• Reduce num_flow_timesteps from 8 to 4 (lower latency)
• Verify camera framerate matches training FPS (LeRobot default: 30fps)
• Enable --policy.use_cache=true to reuse KV-cache between timesteps

Serial port not found:

ls /dev/ttyACM* /dev/ttyUSB*
# Add user to dialout group
sudo usermod -a -G dialout $USER

For deployment details after fine-tuning, see PEFT for VLA: LoRA Deploy on Real Robots.

MolmoAct2 vs DM0.5/OpenDM

Both support SO-101 + LoRA + LeRobot. Key differences:

MolmoAct2 leads on real-world robustness and openness; DM0.5 has simpler out-of-the-box setup. See Fine-tune DM0.5/OpenDM on SO-101 for the equivalent pipeline.

Summary

MolmoAct2 is worth trying for three concrete reasons: (1) fully open-source including training code and datasets, (2) real-world OOD results significantly exceed π0.5, and (3) LeRobot v0.6 makes LoRA fine-tuning on a 24GB GPU straightforward with a single command. With a SO-101 and a capable GPU, the entire pipeline from data collection to inference can run in a day.

Next step: add reward model evaluation for automated trajectory quality scoring — see LeRobot v0.6: Reward Models and lerobot-eval CLI.

Related Posts

• LeRobot Framework: VLA Basics from Scratch — foundations before fine-tuning
• Fine-tune DM0.5/OpenDM on SO-101 — another VLA on the same hardware
• PEFT for VLA: LoRA Deploy on Real Robots — what to do after training completes