EgoHumanoid: Train Humanoid VLA from Human VR Demo

Robot teleoperation data collection has always been the bottleneck of every VLA pipeline. Each episode requires a functioning robot, a skilled operator, and substantial setup time — averaging 62.1 seconds per demo. Scaled across thousands of episodes, the cost is enormous, and the robot's skills remain confined to one or a few fixed environments.

EgoHumanoid — from OpenDriveLab and MMLab @ HKU, accepted at Robotics: Science and Systems (RSS) 2026 — asks a different question: can we replace most robot teleoperation with egocentric demonstrations from humans wearing VR headsets?

The answer is yes, and the results exceed expectations: co-training with human demos achieves 82% success rate in never-before-seen environments, outperforming the robot-only baseline by 51 percentage points. The full codebase is public at github.com/OpenDriveLab/EgoHumanoid under Apache 2.0.

EgoHumanoid overview — source: arXiv:2602.10106, OpenDriveLab & MMLab @ HKU

Why Can Humans Teach Robots?

Humanoid robots are designed to operate like humans — bipedal locomotion, bimanual manipulation, egocentric camera mounted on the head at a similar vantage point. The Unitree G1 stands 1.3m tall with a ZED X Mini mounted on its head. A human wearing a VR headset stands ~1.7m tall with a similar camera on top.

When a person walks into a room, observes through an egocentric camera, and manipulates objects, that data closely resembles what a G1 would see performing the same task. Only three gaps need to be bridged:

1. View gap — human camera at ~1.7m vs G1 at ~1.3m; different perspective and angles
2. Action gap — human hand motions need to be mapped to robot end-effector commands
3. Navigation gap — humans move naturally without explicit locomotion command labels

EgoHumanoid addresses all three gaps through automated view alignment, action alignment, and locomotion command extraction — no manual annotation required.

Hardware Setup: PICO VR + ZED X Mini

Collecting human demonstrations

The demonstrator wears a PICO VR headset equipped with 5 motion trackers that capture full-body pose (24 body keypoints + 26 hand keypoints per hand). A ZED X Mini stereo camera mounted on the headset records egocentric RGB at 960×540 resolution, 20Hz. Handheld PICO controllers track 6-DoF wrist pose; the controller trigger controls the Dex3 dexterous hand.

Each human demo episode captures: synchronized egocentric RGB + 6-DoF wrist poses + navigation intent (forward/back, lateral, yaw, stand/squat) + grasp state. Collection speed: 39.7 seconds/episode — nearly twice as fast as robot teleoperation (62.1 seconds/episode).

Collecting robot demonstrations

G1 is teleoperated remotely using a PICO VR headset + handheld controllers — the operator views a live ZED stream from G1's perspective. The same data format is recorded but requires more effort: robot setup, latency handling, and workspace safety. The final dataset mixes both robot and aligned human data converted to LeRobot format.

View Alignment: Closing the Perspective Gap

This is the most technically complex challenge. The camera is at 1.7m for humans and 1.3m for G1. Naively mixing images from both sources confuses the model — the same room looks significantly different from two different heights, with different perspective distortions, different floor visibility, and different object scales.

The view alignment pipeline solves this in three steps:

Step 1 — Depth estimation with MoGe: MoGe (Monocular Geometric Estimator) processes each egocentric frame to estimate an affine-invariant depth map. MoGe requires no camera calibration and generalizes well to diverse indoor environments — critical when human demos are collected across many different rooms.

Step 2 — Point cloud reprojection: Using the depth map and camera intrinsics, the pipeline reconstructs a 3D point cloud of the scene. It then translates the virtual camera 0.25m downward (from human height to robot height) and reprojects the point cloud to a new 2D frame using G1's camera intrinsics. During training, Gaussian noise of ±0.05m is added to the height offset to make the model more robust.

Step 3 — Inpainting missing regions: After reprojection, some image regions become empty (areas the robot camera would see that weren't in the human's field of view at lower height). Stable Diffusion 2.0 (20 denoising steps, classifier-free guidance scale 7.5) fills these regions using surrounding context. The result is an image with the robot's viewpoint but the human's scene semantics.

Action Alignment: Mapping Human Motion to Robot Actions

Since humans aren't controlling a robot, there are no robot action labels. EgoHumanoid infers equivalent actions from wrist pose tracking and body keypoints:

Upper body — Delta end-effector (6-DoF per arm)

Wrist pose 6-DoF from PICO controllers is processed into delta end-effector commands:

# Pseudocode for upper body action alignment
wrist_poses = load_pico_controller_data(episode)  # 100Hz raw

# Smooth with Savitzky-Golay filter to remove human hand jitter
smoothed = savgol_filter(
    wrist_poses,
    window_length=11,   # 110ms window
    polyorder=3,        # cubic — smooth without over-flattening
    axis=0
)

# Compute delta EEF in SO(3) using log/exp maps for correct interpolation
delta_translation = smoothed[1:] - smoothed[:-1]
delta_rotation = SO3.log(SO3.exp(smoothed[:-1]).inv() * SO3.exp(smoothed[1:]))

# Downsample 100Hz → 20Hz (G1 control frequency)
delta_eef = downsample(np.cat([delta_translation, delta_rotation], axis=-1), factor=5)

Savitzky-Golay with window 11 (110ms) and degree 3 effectively removes hand jitter. Human hands oscillate at ~5-10Hz during manipulation; G1's low-level controller cannot track that, so filtering is essential before alignment.

Lower body — Discretized locomotion commands

Humans move continuously without discrete velocity commands. EgoHumanoid discretizes locomotion into 3-bin representations:

Classification is derived from body keypoint velocity using the motion trackers. Height binary (stand/squat) is determined by pelvis height thresholding. This discretization is intentional — G1's locomotion controller accepts bin commands, not raw velocity.

Gripper control

Binary open/close state from finger curvature of the Dex3 hand. The threshold is calibrated per demonstrator since natural "open" curvature varies between individuals.

Policy Model: Fine-tuning π₀.₅

After human demos pass through view + action alignment and are converted to LeRobot format via data_alignment/convert_to_lerobot.py, the pipeline co-trains π₀.₅ (pi-zero-5, the open-source VLA from Physical Intelligence) on the mixed dataset:

Action output architecture

18-dimensional action vector at each step:

[0:6]   → left arm delta EEF  (3D translation + 3D rotation)
[6:12]  → right arm delta EEF (3D translation + 3D rotation)
[12:14] → locomotion          (forward velocity, lateral velocity)
[14]    → yaw rotation
[15:17] → gripper state       (left, right) — binary open/close
[17]    → height delta        (stand/squat)

Action chunk size = 50 steps (~2.5 seconds at 20Hz) — the model predicts 50 consecutive actions per inference call, enabling smooth trajectories and reducing latency bottlenecks.

Training configuration

VRAM requirements

For RTX 4090 (24GB), LoRA is the practical choice. For A100/H100 80GB, full fine-tuning is possible. The paper uses 8× A100 for co-training, but inference can run on a single capable consumer GPU.

Installation and Getting Started

# Clone the repository
git clone https://github.com/OpenDriveLab/EgoHumanoid.git
cd EgoHumanoid

# Install dependencies (Python 3.10, conda env recommended)
pip install -e .

# Download sample dataset from HuggingFace
# (See README for exact dataset links — includes both robot and human demos)

# STEP 1: Run the full human data alignment pipeline
bash data_alignment/human_data_process/run_human_data_pipeline.sh \
  --input_dir /path/to/raw_human_demos \
  --output_dir /path/to/aligned_demos

# The pipeline runs sub-steps:
# - view_alignment/viewport_transform_batch_h5.py  (MoGe depth → reproject)
# - view_alignment/cache_3d.py                     (cache depth computations)
# - action_alignment/process_navigation_pipeline.py (locomotion command extraction)
# - action_alignment/add_hand_status.py             (gripper labels from finger curvature)

# STEP 2: Convert to LeRobot format (mix robot + human data)
python data_alignment/convert_to_lerobot.py \
  --input_dir /path/to/aligned_demos \
  --output_dir /path/to/lerobot_dataset \
  --robot_data_dir /path/to/robot_demos

# STEP 3: Fine-tune π₀.₅ (requires high-VRAM GPU)
python scripts/train.py \
  --config configs/egohumanoid_g1.yaml \
  --dataset_path /path/to/lerobot_dataset

# STEP 4: Serve the policy (inference server)
python scripts/serve_policy.py \
  --checkpoint /path/to/checkpoint \
  --port 8000

# STEP 5: Deploy on G1 (run on the robot)
python scripts/deploy.py \
  --policy_server http://<server_ip>:8000 \
  --robot_ip <g1_ip>

Important note: View alignment (MoGe + SD 2.0 inpainting) takes approximately 2-3 minutes per episode on an A100. For 1,000 human demo episodes, budget 40-50 GPU-hours just for preprocessing — plan your compute accordingly before starting a large collection run.

Results: 51% Improvement in Unseen Environments

In-domain performance (environments seen during training)

Unseen environment generalization

When tested in rooms and setups never seen during training, the robot-only baseline nearly collapses (31%), while co-training maintains 82%. The explanation is direct: human demos collected across many different rooms provide rich visual diversity → VLA learns scene-agnostic features → generalizes well to new environments.

Task-by-task breakdown: 4 loco-manipulation tasks

Four tasks are evaluated on real G1 hardware: pillow placement on sofa, trash disposal into bin, toy transfer across surfaces, and cart stowing. Each task has 2-4 sequential subtasks of increasing difficulty. Co-training reaches 100% on navigation-dominant subtasks (walking to target, repositioning) and 50-60% on precision manipulation subtasks (precise grasping, controlled placement).

Key insight: navigation skills transfer almost perfectly from human data alone, while manipulation precision remains limited by the action alignment gap between human hands and the Dex3 dexterous hand.

Optimal data mixing ratios

Rule of thumb: use more robot data when precision manipulation is the bottleneck; use more human data when navigation diversity is the bottleneck.

Scene diversity beats volume

Increasing human demo collection from 1 → 3 → 5 distinct rooms: unseen environment success rate increases monotonically from ~57% → ~70% → ~82%. 100 episodes across 5 rooms beats 500 episodes in a single room — a critical scalability insight for planning data collection campaigns.

EgoHumanoid vs. Other Data Collection Approaches

EgoHumanoid's unique advantage: anyone wearing a PICO VR headset and moving around a room creates valid training data — no robotics expertise required, no special workspace needed.

Limitations to Keep in Mind

1. Manipulation precision ceiling: Subtasks requiring precision below ~5mm (placing small objects into tight openings) only reach ~50%. The action alignment residual error between human hands and Dex3 is still a bottleneck.

2. Compute-intensive preprocessing: View alignment (MoGe + SD 2.0 inpainting) costs 2-3 minutes/episode on an A100. 1,000 human demo episodes require ~40-50 GPU-hours just for preprocessing.

3. Dex3 hand dependency: The dataset and pipeline are optimized for the Dex3 hand. Adapting to other grippers (Inspire, Shadow, parallel-jaw) requires re-calibrating the action alignment module.

4. Full fine-tuning needs >70GB VRAM: Only A100 80GB or H100 80GB can run full fine-tuning. For more accessible hardware, LoRA is the practical path, though results may be slightly lower.

5. G1-specific validation only: The paper does not test cross-embodiment transfer. Moving to other humanoids (Fourier GR2, Booster T1, etc.) requires re-calibrating the full alignment pipeline due to different camera positions and action spaces.

Why EgoHumanoid Matters for Small Labs and Startups

A robotics lab or startup with 1-2 humanoid robots and limited teleoperation expertise faces a real chicken-and-egg problem: can't collect enough data without skilled operators, can't afford skilled operators without demonstrating capability first.

EgoHumanoid breaks this loop:

• Low hardware cost: PICO 4 Enterprise ($1,000) + ZED X Mini ($400) vs. continuous G1 operation costs
• No expertise barrier: Any team member after 30 minutes of training can produce useful demonstrations
• Scalable diversity: 10 people demonstrating in 20 different rooms creates 200× more scene diversity than 1 robot in 1 room
• Zero robot risk: Human demos don't break hardware, have no downtime, and require no workspace clearance

The full pipeline is open-source under Apache 2.0 — ready for production use.

Conclusion

EgoHumanoid establishes an important precedent: the data bottleneck in whole-body VLA training does not have to be robot teleoperation. With a sufficiently good alignment framework, egocentric human demonstrations — cheap, diverse, and scalable — can replace the majority of robot data and significantly improve generalization to unseen environments.

This represents a meaningful shift in how the robotics community should think about the data flywheel. Instead of "more robots → more data → better robots," EgoHumanoid shows the loop can be bootstrapped with inexpensive and diverse human data, then progressively enriched with robot data as the pipeline matures.

Original paper: EgoHumanoid: Unlocking In-the-Wild Loco-Manipulation with Robot-Free Egocentric Demonstration — Modi Shi, Shijia Peng, Jin Chen, Haoran Jiang, Tianyu Li, Di Huang, Ping Luo, Hongyang Li, Li Chen — OpenDriveLab & MMLab @ HKU, RSS 2026.

Related Posts

• OpenWBT: Whole-Body Teleoperation for G1 in MuJoCo and Isaac — The traditional starting point: full robot teleoperation before human demos
• EgoHumanoid Step-by-Step Lab Series — Detailed code walkthrough of the EgoHumanoid pipeline
• OpenWBC: Unitree G1 VR Teleop Whole-Body VLA — A parallel approach: optimizing VR teleoperation rather than using human demos