LingBot-VLA 2.0: Scaling VLA to 60,000 Hours of Data and Cross-Embodiment Mobile Manipulation

Paper: From Foundation to Application: Improving VLA Models in Practice — Robbyant (Ant Group), arXiv 2607.06403, July 2026
Code: github.com/Robbyant/lingbot-vla-v2 — Apache 2.0
Checkpoint: HuggingFace robbyant/lingbot-vla-v2

The Problem: Why VLA Models Fail in the Real World

You've probably seen it: a VLA model that looks perfect in the lab, smooth demo on a Franka 7DoF, impressive benchmark numbers — and then fails the moment you put it on a real production robot. This is the lab-to-real gap, robotics' most persistent headache.

Three root causes explain most failures:

1. Action space is too narrow — most models only understand dual-arm 7DoF. Real robots have mobile bases, waists, dexterous hands.
2. Data isn't diverse enough — training on one robot type, deploying on another causes severe distribution shift.
3. No temporal reasoning — models "see current state → generate action" with no prediction of future consequences.

LingBot-VLA 2.0 from Robbyant (Ant Group's robotics AI division) directly addresses all three: 60,000 hours of real-world data, a unified 55-dimensional action space spanning 20 robot configurations, and a novel Dual-Query Distillation mechanism that teaches the model to anticipate future states.

Overview: What Is LingBot-VLA 2.0?

LingBot-VLA 2.0 framework — source: Robbyant/lingbot-vla-v2 repo

LingBot-VLA 2.0 is a 6-billion-parameter Vision-Language-Action foundation model, open-sourced under Apache 2.0. It's designed for cross-embodiment control — a single model that can drive 20+ different robot morphologies, from a simple single-arm Franka to a full humanoid Unitree G1 with 32DoF.

Key specifications:

Compared to version 1.0 (released January 2026), version 2.0 delivers three core upgrades:

• Entirely rebuilt data pipeline — 60K hours vs ~10K hours, with rigorous quality filtering
• MoE action expert — sparse Mixture-of-Experts replacing the dense action decoder
• Dual-Query Distillation — novel technique enabling predictive, anticipatory behavior

Part 1: 60,000 Hours of Training Data — How?

To put 60,000 hours in context: π0.5 (Physical Intelligence) uses ~10,000 hours; GR00T N1.7 (NVIDIA) uses ~22,000 hours. LingBot-VLA 2.0 is roughly 6× larger than π0.5's dataset. The critical question isn't just scale — it's quality.

Dataset composition

The 20 robot configurations include:

• Single-arm: Franka Panda, Flexiv Rizon 4
• Dual-arm: AgileX Cobot Magic, ARX Lift2, UR7e
• Half-humanoid: AgiBot G1, Galbot G1, Astribot S1 (up to 25DoF)
• Full humanoid: Unitree G1, Fourier GR-2, AgiBot A2 (26-32DoF)

Data processing pipeline

This is where most teams cut corners — raw data collection isn't enough. The quality filtering pipeline is what separates useful training data from noise.

For robot trajectory data — four filtering stages:

Stage 1: Jerk-based trajectory smoothness check
  → Compute jerk (3rd derivative of position w.r.t. time)
  → Calculate Z-score of jerk across full trajectory
  → Reject segments with anomalous jerk values
    (catching: robot slips, hardware lag spikes, recording artifacts)

Stage 2: State replay validation
  → Replay action sequence in a physics simulator
  → Compare resulting state trajectory against recorded trajectory
  → Reject if deviation exceeds threshold
    (catching: timestamp drift, logging synchronization bugs)

Stage 3: Human verification sampling
  → Randomly sample 5% of data for manual review
  → Verify video frame aligns with robot state at each timestep
  → Ensure no timestamp mismatches between modalities

Stage 4: Minimum quality threshold
  → Keep only clips with ≥ 20% valid frame ratio
  → Discard clips that are too short or artifact-heavy

For egocentric human video — a more complex pipeline:

Human video doesn't come with joint angles or end-effector positions. To transform it into robot training data, the team applies a computer vision pipeline:

Step 1: VLM pre-filtering
  → Use a vision-language model to classify video content
  → Keep only manipulation sequences
  → Discard: eating, walking, writing, irrelevant content

Step 2: SLAM camera pose estimation
  → Compute camera trajectory in 3D world space
  → Establish reference frame for all video frames

Step 3: MANO hand pose recovery
  → Fit MANO hand mesh model to each frame
  → Extract 3D joint positions for fingers and wrist

Step 4: World-frame trajectory lifting
  → Transform hand poses from camera frame to world frame
  → Map to robot action space

Step 5: Final quality control
  → Physiological constraint checking (joint angle limits)
  → Reject clips with tracking failure
  → Minimum 20% valid frame ratio

The key insight: human video becomes "free robot data" — no teleoperators needed, no robot hardware required. This is how Robbyant scaled to 60K hours without a massive data collection fleet.

Part 2: Unified 55D Action Space — A Common Language for 20 Robots

The fundamental challenge of cross-embodiment control: every robot has a different structure. Franka: 7 joints. Unitree G1: 23 joints. AgileX Cobot Magic: 14DoF dual-arm. How do you train one model to control all of them?

The solution: map everything into a shared 55-dimensional canonical vector.

The elegant design: robots missing a component simply have those dimensions set to zero. Single-arm robots zero out the right arm. Robots without a mobile base zero out the 3D mobility dimensions. The model learns from data which dimensions to activate for each embodiment type.

Relative vs. absolute positions — ablation results:

The choice of action representation has a large impact on performance. The team tested relative joint positions (delta from current state) versus absolute positions (world-frame pose):

Relative positions win decisively. Why? They reduce dependency on global calibration — the robot only needs to know "how much to move," not "where exactly am I in world space."

MeanStd normalization — why not MinMax:

About 10% of robot trajectories contain corrective motions — the robot catching a slip, re-gripping an object, making a large corrective move. These have amplitude > 1.5σ above the mean. MinMax normalization compresses these values toward zero, destroying exactly the information needed for robust manipulation. MeanStd normalization with ±3σ clipping preserves these critical edge cases.

Part 3: MoE Action Expert — Smarter Scaling

Instead of a dense action decoder, LingBot-VLA 2.0 uses a token-level sparse Mixture-of-Experts for the action generation head.

The MoE formulation:

m_ℓ(u) = E_shared(u) + λ · Σⱼ g_j(u) · E_routed_j(u)

Where:
- E_shared(u)    : Shared expert capturing universal control priors
                  (applies across all robots and tasks)
- E_routed_j(u) : N specialized routed experts
                  (some focus on arm control, others on mobile base, etc.)
- g_j(u)        : Sigmoid-based routing weights
                  (NOT softmax — allows multi-expert activation per token)
- λ              : Scaling factor for routed expert contributions

Why sigmoid routing instead of softmax (top-K)?

Standard top-K routing forces each token to hard-select K experts and ignore all others. Sigmoid routing allows each token to receive a blended contribution from multiple experts at varying intensities. For cross-embodiment control, an action token often needs to blend knowledge from multiple robot types — sigmoid routing is fundamentally better suited to this.

Load balancing without auxiliary loss:

The classic MoE problem: without constraints, models collapse to using a few "favorite" experts, wasting capacity. The traditional fix is an auxiliary load-balancing loss, but this creates a trade-off with the primary task objective.

LingBot-VLA 2.0 uses routing correction biases (inspired by DeepSeek-V3): automatically adjusting gate biases to balance load across experts without any explicit loss term.

Ablation results confirm the advantage:

MoE achieves both lower training loss and lower validation error under the same compute budget.

Part 4: Dual-Query Distillation — Teaching Robots to Anticipate

This is the most technically novel contribution of LingBot-VLA 2.0. The core idea: instead of "observe current state → generate action," train the model to simultaneously understand scene geometry and future visual dynamics.

Two learnable query vectors:

Q_t (Current Query):
  → Appended to visual + text tokens
  → Supervised by LingBot-Depth teacher model
  → Loss: ||Proj_depth(Q_t) - D_t||₁
  → Learns: 3D geometry of the current scene (depth map)
  → Effect: robot knows precise spatial distances to objects

Q_{t+T} (Future Query):
  → Appended after Q_t
  → Supervised by DINO-Video teacher model
  → Loss: ||Proj_video(Q_t) - Z_t||_F²   (Frobenius norm)
  → Learns: visual representation of the scene T timesteps later
  → Effect: robot "imagines" what the scene will look like after the action

What is DINO-Video?

Robbyant built DINO-Video specifically for this purpose — a video representation model:

• Base: DINOv3 extended with causal temporal attention
• Temporal attention: Causal (attends only to past frames, not future — critical for real-time use)
• Position encoding: 3D-RoPE (Rotary Position Embedding) covering spatial + temporal axes
• Training data: 5 million video clips
• Output: Feature vector Z_t representing visual state at time t+T

Why two teachers instead of one?

The two teachers provide complementary information that neither can supply alone:

• LingBot-Depth (geometric teacher): tells the model where things are — crucial for grasping precision and avoiding collisions
• DINO-Video (temporal teacher): tells the model how things will change — crucial for long-horizon planning and preemptive error correction

The combined effect: the model develops genuine anticipatory behavior. It adjusts its trajectory before getting stuck, rather than after.

Installation and Setup

System requirements

# Clone the repository
git clone https://github.com/Robbyant/lingbot-vla-v2
cd lingbot-vla-v2

# System requirements:
# - Python 3.12
# - PyTorch 2.8.0
# - Flash-attn 2.8.3
# - CUDA 12.x
# - RAM: ≥ 64GB recommended
# - GPU VRAM: ≥ 24GB for inference, ≥ 80GB×8 for full training

# Create conda environment (automated script)
bash tools/create_train_env.sh
conda activate lingbot-vla-v2

Download model weights

# Main checkpoint (6B params)
huggingface-cli download robbyant/lingbot-vla-v2-6b \
    --local-dir ./checkpoints/lingbot-vla-v2-6b

# Vision-language backbone (required)
huggingface-cli download Qwen/Qwen3-VL-4B-Instruct \
    --local-dir ./checkpoints/Qwen3-VL-4B-Instruct

# Depth encoder for Dual-Query Distillation
huggingface-cli download robbyant/MoGe-2-vitb-normal \
    --local-dir ./checkpoints/MoGe-2-vitb-normal

Fine-tuning on Your Own Robot Data

LingBot-VLA 2.0 uses LeRobot v2.1/v3.0 format — the same as most modern VLA frameworks, making integration straightforward.

Step 1: Prepare your dataset

# Your dataset needs to be in LeRobot format.
# See: https://github.com/huggingface/lerobot for conversion guides.

# Expected directory structure:
# data/
#   my_robot_dataset/
#     meta/
#       info.json          # Dataset metadata
#       episodes.jsonl     # Episode list
#     videos/              # Camera recordings
#     data/                # Robot states + actions

Step 2: Define your robot config

# configs/robots/my_robot.yaml
# Maps your robot's features to the unified 55D action space

robot_name: "my_franka_robot"
embodiment_type: "single_arm"  # single_arm | dual_arm | half_humanoid | humanoid

feature_mapping:
  # Left arm joints (7DoF Franka)
  left_arm_joints:
    source_keys: ["observation.state.joint_positions"]
    target_indices: [0, 1, 2, 3, 4, 5, 6]  # First 7D of 55D vector
    representation: "relative"              # Use relative (delta) positions

  # Gripper
  left_gripper:
    source_keys: ["observation.state.gripper_pos"]
    target_indices: [28]                    # Index 28 = left gripper in 55D
    representation: "absolute"

  # All unused dimensions (right arm, waist, head, mobile base) default to 0

normalization:
  type: "meanstd"
  clip_ratio: 3.0   # Clip at ±3σ to preserve corrective motions

Step 3: Compute normalization statistics

python scripts/compute_norm_stats.py \
    --dataset-path ./data/my_robot_dataset \
    --robot-config configs/robots/my_robot.yaml \
    --output-path ./data/my_robot_dataset/norm_stats.json

Step 4: Run fine-tuning

# Multi-GPU training (recommended for production)
torchrun --nproc_per_node=8 tasks/vla/train.py \
    --config configs/train/lingbot_vla_v2_finetune.yaml \
    --dataset-path ./data/my_robot_dataset \
    --checkpoint ./checkpoints/lingbot-vla-v2-6b \
    --output-dir ./outputs/my_finetune

# Single GPU (experimental / debugging only)
python tasks/vla/train.py \
    --config configs/train/lingbot_vla_v2_finetune.yaml \
    --dataset-path ./data/my_robot_dataset \
    --checkpoint ./checkpoints/lingbot-vla-v2-6b \
    --num-gpus 1 --batch-size 4

Important training config options:

# configs/train/lingbot_vla_v2_finetune.yaml
training:
  # Sequence-wise auxiliary loss — significantly improves training stability
  sequence_wise_auxiliary_loss: true

  # MoE routing regularization — prevents expert collapse
  z_loss_weight: 0.001

  # Optional: Muon optimizer (experimental, may be unstable)
  muon_optimizer: false

  # Learning rate
  lr: 2e-5
  warmup_steps: 100

  # Diffusion steps — more = slower but more accurate
  num_inference_steps: 10   # 10 for production, 20 for higher accuracy

Real-Robot Inference

from deploy.lingbot_vla_v2_policy import LingBotVLA2Policy
import numpy as np

# Load the fine-tuned model
policy = LingBotVLA2Policy(
    checkpoint_path="./outputs/my_finetune/checkpoint_final",
    robot_config="configs/robots/my_robot.yaml",
    device="cuda:0"
)

def run_control_loop(robot, camera, instruction):
    """
    observation dict expects:
    - "image": numpy array (H, W, 3)
    - "instruction": natural language string
    - "robot_state": current joint positions / gripper state
    """
    while not task_complete(robot):
        obs = {
            "image": camera.get_frame(),
            "instruction": instruction,
            "robot_state": robot.get_state()
        }

        # Inference: ~130ms on RTX 4090D
        action_55d = policy.get_action(obs)

        # Decode 55D → actual robot commands via robot_config mapping
        robot_commands = policy.decode_action(action_55d)
        robot.execute(robot_commands)

# Example usage
run_control_loop(
    robot=my_robot,
    camera=my_wrist_camera,
    instruction="Pick up the red cup and place it on the tray"
)

Latency breakdown on RTX 4090D:

• Visual encoding (Qwen3-VL): ~40ms
• MoE action expert (10 denoising steps): ~70ms
• Overhead (I/O, decoding): ~20ms
• Total: ~130ms → 7-8Hz control loop

For higher-frequency control, reduce to 5 denoising steps (~70ms) with a ~7-10% accuracy trade-off.

Benchmark Results

GM-100 — Bimanual Manipulation Benchmark (Shanghai Jiao Tong University)

GM-100 is a 9-task generalist benchmark covering real-world bimanual tasks including keychain retrieval, toy sorting, wrapping, and tool use.

Format: Task Progress (%) / Success Rate (%)

Standout results: "Retrieve Keychain" hits 100%/100% on AgileX — perfect score. "Pick Out Toy Bones" (fine dexterous manipulation): 87.5%/70%.

Long-Horizon Mobile Manipulation

This is where LingBot-VLA 2.0 truly distinguishes itself — tasks requiring the robot to navigate, locate objects, and manipulate across multiple sequential steps.

Design choice ablation — full summary

Every component contributes meaningfully. The combination of all three — relative positions + MeanStd + MoE — gives the full performance.

Comparison With Other VLA Models

Where LingBot-VLA 2.0 wins: data breadth (most embodiment types), data scale (6× larger than π0.5), and mobile manipulation capability (the only model tested on long-horizon mobile tasks in this benchmark).

Where it requires more work: GPU requirements are steep (RTX 4090 minimum for inference), and training cost is high. Not yet competitive on tasks that need sub-50ms control loops.

Common Pitfalls When Deploying

1. Robot config mapping errors are silent and dangerous

If the feature mapping from your robot to 55D space is wrong, the model outputs plausible-looking but incorrect actions — no error messages, no obvious failure mode. Always log the raw 55D output vector and verify which dimensions are being activated.

2. Always recompute normalization statistics after fine-tuning

Never reuse the pretrained checkpoint's norm stats after fine-tuning on your own data. The action distribution shifts when you specialize the model — stale norm stats will cause systematic bias in action magnitudes.

3. GPU memory requirements

6B params + activations + optimizer states: full training needs at least 80GB VRAM (8×A100 recommended). For 24GB GPUs (RTX 4090), inference is fine, but full fine-tuning requires LoRA with gradient checkpointing.

4. Mobile base timing synchronization

When your robot has a mobile base: the 3D mobility dimensions (indices 52-54) command the base while arm joints command the arms simultaneously. These commands must be synchronized with < 50ms error — larger drift causes visible coordination failures during locomotion-while-grasping tasks.

5. Diffusion step count trade-off

5 steps  → ~70ms  → ~7-10% accuracy drop  (acceptable for fast-reaction tasks)
10 steps → ~130ms → Baseline accuracy     (recommended for general use)
20 steps → ~250ms → ~2-3% gain            (diminishing returns, rarely worth it)

Conclusion

LingBot-VLA 2.0 sets a new standard for data scale and embodiment coverage in the 2026 VLA landscape. The three core technical contributions — Unified 55D Action Space, MoE Action Expert, and Dual-Query Distillation — each address a specific practical failure mode that previous VLA models left unsolved.

The most impressive aspect isn't any single benchmark number. It's the design philosophy: rather than optimizing for one specific robot or task type, Robbyant invested in the infrastructure (data pipeline, unified action representation) that enables natural scaling. That's what separates a genuine foundation model from a task-specific policy.

With Apache 2.0 licensing, a public 6B checkpoint, and a codebase built around the widely-adopted LeRobot format, LingBot-VLA 2.0 is the most accessible large-scale cross-embodiment VLA released as of mid-2026.

Paper: arXiv 2607.06403 — From Foundation to Application
Code: github.com/Robbyant/lingbot-vla-v2
Checkpoint: HuggingFace robbyant/lingbot-vla-v2-6b

Related Posts

• LingBot-VA: Causal World Model for Robot Manipulation
• VLA Mobile Manipulation with LeRobot — Practical Guide
• HEX-VLA: Cross-Embodiment Humanoid Whole-Body Control