DyPES-VLA: Learning Shared Dynamics Priors and Embodiment-Specific Control for Cross-Embodiment Manipulation

Imagine teaching three students simultaneously: one with short arms, one with long arms, and one with two sets of hands. Same lesson, same brain — but the way each holds the pen is completely different.

That's precisely the cross-embodiment manipulation problem in robotics: how do you train a single policy that can control a 7-DoF Franka Panda, a 29-DoF Unitree G1 humanoid, and a 14-DoF dual-arm robot — using the same model weights?

DyPES-VLA (arXiv 2608.06374, August 2026 — HKUST Guangzhou & COCO Matrix) answers this with two key innovations:

1. Dynamics Prior: Train the VLM to learn the "physics" of manipulation from action-free videos — how objects move, how hands make contact, how scenes change after manipulation.
2. Mixture-of-Experts (MoE) Action Head: Each embodiment gets its own specialized FFN expert — no manual action space alignment, no preprocessing pipelines.

Results: 98.0% on LIBERO, 89.02% on RoboTwin 2.0, 59.25% on RoboCasa-GR1, and 75.6% real-world across 3 physical robots — outperforming GR00T-N1.6 (59.6%) by +16 points.

Why Is Cross-Embodiment Manipulation Hard?

Most current VLAs — from Qwen-VLA to X-VLA — are trained on a single robot type or require manual action space normalization (e.g., mapping everything to Cartesian end-effector space). This creates two fundamental problems:

Problem 1 — Wasted shared knowledge: Every robot obeys the same physics. When you pull a box off a shelf, whether you use a human hand, a Franka arm, or a humanoid hand, the box moves the same way. Yet current models train action policies from scratch per embodiment, wasting this shared "physics knowledge."

Problem 2 — Manual preprocessing burden: Converting 29-DoF humanoid joint-space actions to 7-DoF Franka end-effector Cartesian is non-trivial. It requires complex IK/FK pipelines that introduce errors and information loss.

DyPES-VLA cleanly separates two concerns:

• Learning "shared physics" (Dynamics Prior) → uses video data, no robot action labels needed
• Learning "how to use that physics per robot" (MoE Action Head) → embodiment-specific, no preprocessing needed

Architecture Overview

The model has three main components:

1. Vision-Language Model with Shared Query Tokens

The backbone is Qwen3-VL-2B, encoding:

• Visual observations (camera images)
• Language instructions ("pick up the red cube")
• Embodiment metadata (robot type, action format)
• N=96 learnable query tokens — the critical shared interface

These 96 query tokens are trained to capture shared dynamics: object motion, hand-object contact patterns, scene changes after manipulation. They form a "common language" that Franka, humanoid, and dual-arm robots all share.

2. Future Generation Head (Dynamics Prior)

The core innovation. Instead of learning only from robot demonstration data, the model is trained to predict future frames — a task requiring no robot action labels.

The generator uses SANA-600M (image diffusion model) with a rectified-flow objective:

$$\mathcal{L}\text{future} = \mathbb{E}{\tau, \epsilon}\left[|g_\psi(z_\tau, \tau, p_\omega(Z)) - (z - \epsilon)|_2^2\right]$$

Where:

• $z_\tau$ is the noisy latent of the future frame
• $p_\omega(Z)$ are query states from the VLM (96 tokens)
• $g_\psi$ is SANA-600M conditioned on query states

The goal: force the 96 query tokens to encode sufficient information to predict the future — thereby naturally learning manipulation dynamics.

3. Embodiment-Specific MoE Action Head

This component translates "shared dynamics language" into "per-robot commands." The design is a 16-layer Diffusion Transformer with K=3 experts:

• Per-Embodiment Encoder-Decoder: Lightweight adapters mapping each robot's native action space into/out of a common latent
• Shared Attention Layers: Cross-attention to the 96 query tokens; self-attention over the action sequence — both using Adaptive LayerNorm (AdaLN). Shared across all embodiments
• Static Router: Selects one of K=3 FFN experts based on embodiment metadata — no complex learned gating needed
• Embodiment-Specific FFN Experts: Each expert handles the unique kinematic constraints of one embodiment

Stage 2 training objective:

$$\mathcal{L} = \mathcal{L}\text{action} + \lambda_w \mathcal{L}\text{future}, \quad \lambda_w = 0.05$$

The 0.05 weight keeps balance: action loss dominates, future loss prevents dynamics prior forgetting.

Two-Stage Training in Detail

Stage 1: Dynamics Prior Pretraining (100K steps)

Goal: Learn shared manipulation dynamics from videos — no robot action labels required.

Hardware: 16 H100 GPUs, batch size 512.

Data mixture:

Why 50% EgoDex? Human hand manipulation videos contain the richest real-world dynamics — grasping physics, contact patterns, and object behavior that transfer well to robots.

Only the Future Generation Head and Query Tokens are trained here. The VLM backbone is lightly fine-tuned. The Action Head is not yet trained.

Stage 2: Co-Training with MoE Action Head (200K steps)

Goal: Train embodiment-specific action policies while maintaining dynamics prior via joint loss.

Data mixture (robot demonstration data only):

Per-embodiment action horizons:

• Single-arm (Franka): 8 steps
• Humanoids: 16 steps
• Dual-arm: 50 steps

Inference: Only 4 Euler steps for flow integration — much faster than traditional DDPM.

Real-World Finetuning (5K steps)

After simulation training, finetune for 3 physical robots:

• FR3 (Franka Research 3)
• COBOT Magic
• Unitree G1 (humanoid)

Only 1,800 demonstrations required — remarkably data-efficient for 3 embodiments.

Benchmark Results

Simulation Benchmarks (single checkpoint — same weights across all 3)

LIBERO (7-DoF Franka Panda):

RoboCasa-GR1 (29-DoF Humanoid):

RoboTwin 2.0 (14-DoF dual-arm):

Real-World Performance

DyPES-VLA outperforms GR00T-N1.6 by +16 points and ACT by +43 points in real-world evaluation.

Ablation Study: How Much Does Each Component Matter?

Key takeaways:

• Future Prediction matters most for humanoid and dual-arm (more complex tasks, harder dynamics)
• MoE matters more for humanoid (29-DoF, kinematic constraints very different from single-arm)
• LIBERO is least affected — it's a simpler single-arm task with less complex dynamics

Implementation Guide

Code is not yet public as of August 2026, but you can implement the approach from the paper:

Environment Setup

# Core dependencies
pip install torch torchvision
pip install transformers  # for Qwen3-VL-2B
pip install diffusers     # for SANA-600M
pip install lerobot       # for LIBERO, RoboTwin data loading

# LIBERO benchmark environment
pip install libero

Stage 1: Dynamics Prior Training

import torch
from transformers import Qwen3VLForConditionalGeneration

# Load Qwen3-VL-2B backbone
vlm = Qwen3VLForConditionalGeneration.from_pretrained("Qwen/Qwen3-VL-2B")

# Add 96 learnable query tokens
query_tokens = torch.nn.Parameter(
    torch.randn(1, 96, vlm.config.hidden_size)
)

# Rectified-flow future prediction loss
def compute_future_loss(query_states, future_frame_latent, noise):
    """
    query_states: [B, 96, D] — VLM output
    future_frame_latent: z — latent of target future frame
    noise: epsilon ~ N(0, I)
    """
    t = torch.rand(query_states.shape[0])  # random timestep
    z_t = (1 - t[:, None, None]) * noise + t[:, None, None] * future_frame_latent
    velocity_pred = sana_generator(z_t, t, conditioning=query_states)
    velocity_target = future_frame_latent - noise
    return ((velocity_pred - velocity_target) ** 2).mean()

Data loading: mix EgoDex (50%) + simulation videos (50%) — only (current_frame, future_frame) pairs, no action labels needed.

Stage 2: MoE Action Head

class MoEActionHead(torch.nn.Module):
    """
    16-layer Diffusion Transformer with K=3 embodiment-specific experts.
    """
    def __init__(self, n_layers=16, n_experts=3, d_model=512):
        super().__init__()
        
        # Shared attention layers (cross + self attention)
        self.shared_attention = torch.nn.ModuleList([
            SharedAttentionBlock(d_model) for _ in range(n_layers)
        ])
        
        # Per-embodiment FFN experts
        self.experts = torch.nn.ModuleList([
            torch.nn.Sequential(
                torch.nn.Linear(d_model, d_model * 4),
                torch.nn.GELU(),
                torch.nn.Linear(d_model * 4, d_model)
            ) for _ in range(n_experts)
        ])
    
    def forward(self, action_noisy, query_states, embodiment_id):
        x = action_noisy
        for layer in self.shared_attention:
            x = layer(x, cross_kv=query_states)  # attend to 96 query tokens
        return self.experts[embodiment_id](x)     # route to correct expert

# Joint training objective
def training_step(batch):
    action_loss = compute_action_loss(batch)
    future_loss = compute_future_loss(batch)
    return action_loss + 0.05 * future_loss

Inference

# 4 Euler steps — fast flow integration
def inference(observation, instruction, embodiment_id, n_steps=4):
    query_states = vlm(observation, instruction)  # [B, 96, D]
    action = torch.randn(batch_size, action_horizon[embodiment_id], action_dim)
    
    dt = 1.0 / n_steps
    for i in range(n_steps):
        t = torch.tensor(i * dt)
        velocity = moe_head(action, query_states, embodiment_id)
        action = action + velocity * dt
    
    return action

# Action horizons per embodiment
ACTION_HORIZONS = {
    "single_arm": 8,   # Franka, COBOT Magic
    "humanoid": 16,    # Unitree G1, GR1
    "dual_arm": 50,    # RoboTwin dual-arm
}

Real-World Finetuning

# Finetune from simulation checkpoint for your robot
python finetune.py \
    --checkpoint path/to/simulation_checkpoint.pt \
    --embodiment_id 0 \                    # 0=FR3, 1=COBOT, 2=G1
    --data path/to/your_demos/ \           # 1800 demos recommended
    --steps 5000 \
    --lr 1e-4

Comparison with Alternative Approaches

Use DyPES-VLA when: You have ≥2 different robot types and want a unified policy, or have large video datasets to leverage for pretraining.

Skip DyPES-VLA when: You only have one robot type — a specialist policy (Diffusion Policy, ACT) is simpler and sufficient.

Three Key Insights from DyPES-VLA

Insight 1: Video data is an underexploited gold mine. EgoDex and manipulation videos exist in abundance — they contain richer dynamics information than robot demo data. DyPES-VLA exploits this through future prediction pretraining. No expensive robot collection needed.

Insight 2: Decouple "physics" from "control." Dynamics Prior learns shared physics, MoE Action Head learns per-robot control. This is a clean inductive bias that aligns with how we actually think about manipulation.

Insight 3: Static routing is sufficient. Instead of complex learned gating (as in LLM-style MoE), simple static routing based on embodiment metadata works just as well — because we're not routing between tasks, only distinguishing embodiments.

Limitations and Future Directions

The paper acknowledges several limitations:

• Limited embodiment scale: 3 embodiments in real-world (FR3, COBOT, G1). Scaling to 10-20 embodiments untested.
• K=3 experts works for 3 embodiments but needs more experts as embodiment count grows.
• RoboCasa-GR1 at 59.25% — humanoid manipulation remains hard, especially for tasks requiring good balance.
• Inference speed: No real-time Hz numbers reported — 4 Euler steps through a 16-layer diffusion Transformer may still be too slow for some applications.

Promising future directions:

• Add sim data (Isaac Lab, MuJoCo) to increase dynamics diversity
• Extend to mobile manipulation (wheel + arm)
• Combine with Diffusion Policy approaches for more efficient action heads

Conclusion

DyPES-VLA is a clear step forward in cross-embodiment manipulation. The key number isn't 98% LIBERO (a relatively easy benchmark for large models) — it's 75.6% real-world across 3 physically different robots with a single checkpoint.

The two core ideas — learning shared dynamics from video via future prediction, and routing to per-embodiment experts — are building blocks the community will continue developing through 2026-2027.

Paper: arXiv 2608.06374 | Project page: DyPES-VLA Release

Related Posts

• Qwen-VLA: Multitask Robot Manipulation on LIBERO with 97.9% Success
• X-VLA ICLR 2026: Cross-Embodiment VLA with LeRobot
• RDT-2: Zero-Shot Cross-Embodiment Bimanual Manipulation