ω-0: Latent Predictive WAM for Humanoid Loco-Manipulation

Picture asking a humanoid robot to walk into the kitchen, bend down to open the refrigerator, and bring back a drink — all in one fluid, uninterrupted motion. This is concurrent loco-manipulation: the robot must walk, balance, adjust posture, and manipulate objects simultaneously, exactly the way humans do it without thinking.

Most robotic systems fail at this because of one fundamental design choice: locomotion and manipulation are engineered separately. The robot stops to grasp, then walks, then stops again. That is not just slow — it is completely impractical in real household environments.

On August 6, 2026, researchers from MARS Lab and HMI Lab published ω-0 (omega-zero) — a Latent Predictive World Action Model that directly addresses this problem at real-world scale. The result: 81.8% success rate across 11 household tasks, far exceeding every VLA and WAM baseline tested.

Paper: ω-0: A Latent Predictive World Action Model for Concurrent Humanoid Loco-Manipulation — arXiv:2608.06375, August 2026. Project page: gentlefress.github.io/OMEGA-0_page

Why Concurrent Loco-Manipulation Is Hard

Before diving into ω-0, it is worth understanding exactly why this problem is considered one of the hardest open challenges in modern robotics.

Problem 1 — Timing conflict: Stable walking requires the arms to counterbalance inertia. When those same arms are trying to follow a separate manipulation trajectory, the two objectives physically conflict. Solving this optimization in real time is extraordinarily difficult.

Problem 2 — Decomposition failure: Most systems split the problem into an "upper body controller" (arms) and a "lower body controller" (legs), then stitch them together via an interface. This creates latency and desynchronization — the robot must stop walking before the arms can safely manipulate.

Problem 3 — VLA models are arm-centric: Current VLA models like π-0.5, GR00T-N1.7, and EgoVLA are trained primarily for stationary manipulation — arms moving while the robot stands still. Adding locomotion breaks their action representation entirely.

Problem 4 — Existing WAMs generate unusable video: Prior World Action Models (Fast-WAM, DiT4DiT) use future video prediction as their training objective. This forces an "video → action" inversion at inference time — slow, unstable, and error-prone.

ω-0 addresses all four problems with a fundamentally different architectural philosophy.

The Core Idea: Latent Prediction, Not Video Generation

Instead of predicting future video frames (expensive, hard to use), ω-0 learns to predict compact future observation embeddings — a small vector capturing future state in latent space.

The auxiliary training objective is:

ℒ_video = ||h^v - y^v_{t+1:t+K}||²₂

Where:

• h^v is the video query output from the Joint Predictor
• y^v_{t+1:t+K} is the embedding from a frozen Wan encoder applied to K future frames

Why does this matter? By predicting embeddings instead of pixels, the model develops "future awareness" at a fraction of the computational cost of video generation. And because this is only an auxiliary task, the primary pathway remains direct action generation — no inversion needed at inference.

In plain terms: ω-0 learns to look ahead without needing to draw the future — it uses that foresight to make better action decisions right now.

Three-Stage Architecture

ω-0 is built as a three-stage pipeline where each stage builds directly on the previous one.

Stage 1: Whole-Body Action VLM

Model: Fine-tuned Qwen3-VL-2B-Instruct with a custom FAST tokenizer.

FAST (Frequency-Adaptive Sequence Tokenizer) converts continuous whole-body trajectories into discrete tokens. The VLM then autoregressively predicts action tokens from:

• Language instruction
• Egocentric RGB (robot-mounted camera)
• Exocentric RGB-D (third-person depth camera)

Stage 1 is trained on large-scale public data: ARCTIC, Xperience-10M, and Motion-X — giving the VLM a general motion prior before it ever sees real robot data.

Stage 2: Joint Video-Action Latent Predictor

This is the heart of ω-0 — the module that fuses multiple information streams:

The module uses two parallel sets of learnable queries:

• Motion queries: learn to generate action representations
• Video queries: learn to predict future visual embeddings

Both use token-specific RoPE (Rotary Position Embedding):

• Visual tokens: 2D RoPE
• Video queries (temporal): 3D RoPE
• Action queries: 1D RoPE

This separation lets each query type "attend" to exactly the right spatial-temporal structure for its role.

Stage 3: Action DiT (Diffusion Transformer)

The Joint Predictor output (motion representation + text + robot state) feeds into an Action DiT — a Diffusion Transformer that denoises whole-body action latents.

Action space: 66 dimensions:

• 64-dim: whole-body action latent for the SONIC controller (resolved into joint commands)
• 2-dim: hand grasp commands [0, 1]

The SONIC controller (from GR00T-N1.5) acts as the low-level motion tracker — receiving action latents from ω-0 and converting them to joint-level torque commands for the physical robot. For details on SONIC, see SONIC Whole-Body Controller on GEAR Humanoid.

The ω-HOME Dataset: 40 Hours of Real Robot Data

The team collected ω-HOME — the largest publicly announced household humanoid manipulation dataset to date.

8 capability groups:

1. Object retrieval
2. Surface cleaning
3. Appliance interaction
4. Container transfer
5. Cloth handling
6. Storage
7. Mobile manipulation
8. Floor operations

Multi-modal data per episode:

• Egocentric RGB (robot head camera)
• Exocentric RGB-D (third-person + depth)
• Whole-body SMPL motions (captured from teleoperation)
• Robot states (joint positions, hand configuration, torso 6D rotation)
• Action latents (64-dim SONIC latents)

Teleoperation setup: An operator wears a motion capture suit; the robot mirrors the human motion via the SONIC motion tracker. This captures genuine whole-body demonstrations — not just arm motions, but coordinated body-arm-leg movements.

Three-Stage Training Pipeline

Stage 1: Whole-Body Action VLM Pretraining

# Conceptual — train FAST tokenizer on continuous trajectories,
# then fine-tune VLM with next-token prediction

# Public data used:
# - ARCTIC (hand-object interaction capture)
# - Xperience-10M (egocentric video + motion)
# - Motion-X (large SMPL motion library)

The goal: give the VLM a broad motion prior from internet-scale data before touching any robot trajectories.

Stage 2: Action-Latent Pretraining with SONIC Simulation Replay

This is ω-0's most novel contribution. The team uses SONIC simulation replay to convert publicly available human motion data into robot-executable supervision.

Human motion datasets (SMPL meshes)
    ↓ SONIC motion tracker (in simulation)
    ↓ Filter physically infeasible motions
    ↓ Convert to 64-dim SONIC action latents
    ↓ Paired with egocentric + exocentric renders
Robot training data (no real teleoperation needed)

Loss function:

ℒ_stage2 = ℒ_action + λ_video · ℒ_video

Where ℒ_video is the future embedding prediction loss using a frozen Wan encoder. The λ_video weight balances the two objectives.

The key insight: ω-0 learns to act like a robot from hours of human video without expensive teleoperation — simulation replay bridges the embodiment gap.

Stage 3: Real-World Fine-Tuning on ω-HOME

Fine-tune the full model on real robot trajectories from ω-HOME. This stage introduces Real-Time Chunking (RTC) — a critical training technique.

What problem does RTC solve? Traditional action chunking predicts N steps but executes from the beginning, creating a pause gap between consecutive chunks that causes jerky motion — unacceptable for continuous locomotion.

RTC trains with overlap between action chunks from the start, so inference can use overlap blending for smooth transitions:

Chunk 1: steps 1→25
  Execute: steps 1→8
Chunk 2 (predicted in parallel): steps 9→33
  Execute: steps 9→16
  Blend: 8 overlap steps for smooth transition
Chunk 3: steps 17→41
  ...

Inference: 7+ Hz on the Real Robot

After training, ω-0 runs with receding-horizon control:

1. Predict a 25-step chunk (~0.14 seconds per forward pass)
2. Execute 8 steps while predicting the next chunk in parallel
3. Cache remaining 17 steps as a prefix for the next prediction
4. Overlap blending at chunk boundaries for smooth transitions

Final control frequency: >7 Hz — sufficient for real-time locomotion on a physical humanoid.

Viewpoint flexibility: At inference time, ω-0 can operate with only the egocentric camera (no external camera required), thanks to learnable view tokens. This is critical for real deployment when a fixed external camera is unavailable.

Results: 81.8% on 11 Household Tasks

Task Suite

11 loco-manipulation household tasks, the majority requiring active lower-body motion:

Baseline Comparison

Gap over best baseline (ψ-0): +37.3 percentage points — an unusually large leap in robotics benchmarking.

Ablation Study

Key takeaways from ablation:

• Proprioceptive state is the single most important input — without it, the model is blind to body configuration during locomotion
• V-JEPA substantially outperforms Wan encoder for temporal visual encoding
• Video queries (future awareness) contribute meaningfully — confirming that latent prediction genuinely improves action quality

Why ω-0 Beats Prior WAMs

To understand ω-0's advantage, compare how prior WAMs handle inference:

Fast-WAM uses video prediction as the primary training signal → inference requires inverting video back to action (slow, error-prone).

DiT4DiT couples two DiTs (video + action) → joint training but still needs test-time video inversion.

ψ-0 (arm-centric) trains a VLA for arms with a separate locomotion controller → two disconnected systems that never learn whole-body coordination.

ω-0 differs fundamentally:

1. Video prediction is auxiliary only — not the primary pathway
2. No video inversion at inference — actions are generated directly
3. The full 66-dim action latent (locomotion + manipulation) is predicted jointly — the model sees whole-body coordination as a single unified problem

Architecturally, ω-0 is closest to GR00T-N1.5/SONIC — both use the SONIC controller — but ω-0 adds the latent prediction layer and a substantially more complex multi-stage training pipeline.

Practical Notes Before Using ω-0

1. Code is not yet public (WIP as of this writing)

The GitHub repository is listed as Work in Progress. Monitor the project page: gentlefress.github.io/OMEGA-0_page

2. Hardware requirements are specific

ω-0 is deployed on humanoid robots with the SONIC whole-body controller. You need:

• A humanoid platform compatible with SONIC (currently Unitree G1 or H1 variants)
• SONIC motion tracker setup
• Egocentric + exocentric cameras (depth for exocentric)

3. Training is multi-stage and compute-intensive

Given the model stack (Qwen3-VL-2B + T5 + V-JEPA + Action DiT):

• Stage 1: Hundreds of GPU-hours on A100/H100 with large public datasets
• Stage 2: SONIC simulation replay requires Isaac Sim or equivalent physics engine
• Stage 3: Fine-tuning on ω-HOME real robot data

4. ω-HOME dataset is coming soon

The 40.3-hour dataset is announced as upcoming. Once released, it will be the most comprehensive real-robot household manipulation dataset available for training similar models.

Technical Summary

Related Posts

• SONIC Controller: Whole-Body Control for GEAR Humanoid — Deep dive into the SONIC controller that ω-0 uses to convert action latents into joint-level commands
• GR00T-SONIC: NVIDIA's Wholebody VLA Pipeline — The direct predecessor architecture that ω-0 builds upon and substantially outperforms
• LeverB: Benchmark for Humanoid Wholebody VLA — Evaluation framework to contextualize ω-0's results against broader humanoid benchmarks