OpenHLM: Empirical Recipe for Humanoid Loco-Manipulation

Imagine asking a robot to pick up an apple from the floor and place it on an overhead shelf. Simple for a human. For a humanoid robot, it demands coordinating 30+ joints — bending the knees, shifting the center of mass, reaching down, grasping, standing up, and then reaching high — all as one fluid motion, with no clean boundary between "leg phase" and "arm phase."

This is whole-body loco-manipulation — and most existing systems dodge it by decoupling locomotion and manipulation into separate controllers. The robot ends up behaving like a wheeled dual-arm platform, not a true humanoid.

A team from Tsinghua University, Shanghai Qi Zhi Institute, and Spirit AI didn't dodge it. They ran systematic, one-variable-at-a-time experiments across three research axes and published the results as OpenHLM (arXiv 2606.22174) — a fully open-source empirical recipe that helps the Unitree G1 outperform both GR00T N1.6 and Ψ₀ while using less than half the total demonstration time.

OpenHLM teaser: Unitree G1 executing loco-manipulation tasks spanning a wide vertical range — source: openhlm-corl.github.io

Why Is Whole-Body Loco-Manipulation So Hard?

Industrial robot arms control 6-7 joints. The Unitree G1 has 29 degrees of freedom across its full body. The challenge: to pick something off the floor and place it on a high shelf, the robot cannot simply extend its arm. It must bend, shift weight, maintain balance, grasp, rise, and reach up — all as a continuous, coupled motion.

Standard solutions decompose this into two separate problems: one controller manages the legs (locomotion), another manages the arms (manipulation). The result looks like a wheeled platform with arms attached — locomotion first, manipulation second, never truly integrated. That's the ceiling of decoupled architectures, and it's where OpenHLM begins.

What Is OpenHLM?

OpenHLM is not a novel architecture or a billion-parameter model. It's a systematic empirical recipe — three independent research threads, each tested with controlled ablations, answering three questions:

1. Teleoperation design — How should a human demonstrate full-body motion to a robot?
2. VLA policy design — Which model architecture handles a 34-DOF action space well?
3. Heterogeneous co-training with HuMI — How do you scale data collection without unlimited robot time?

All code, data, and model checkpoints are publicly available:

• GitHub: github.com/OpenHLM-project/OpenHLM
• Dataset: huggingface.co/datasets/OpenHLM/OpenHLM-data
• Checkpoints: huggingface.co/OpenHLM/OpenHLM-ckpts

Technical Stack: Three Independent Modules

The OpenHLM system is structured as three submodules within a single monorepo:

Required hardware:

• Robot: Unitree G1 + two ChangingTek CTAG2F90-D grippers
• Cameras: Two Intel RealSense D405 (wrist-mounted) + one Unitree SV1-25 fisheye stereo (head-mounted)
• Teleoperation: PICO4U VR kit (HMD + controllers + leg trackers)
• Training GPU: RTX 5080 or equivalent, minimum 8GB VRAM
• OS: Ubuntu 25.04

Phase 1: Whole-Body Teleoperation Design

Three teleoperation interfaces were compared:

Option A — Decoupled (21-DOF): Arms and legs controlled independently. Robot arms mirror operator arms; legs run a separate locomotion controller. Result: no true whole-body coordination.

Option B — VR 3-point (24-DOF): Head + two hands tracked via VR controllers. Simple setup but lacks lower-body pose information.

Option C — Joint-based whole-body (32-DOF): Operator wears a compact motion capture rig tracking full-body pose. Joint angles map directly from operator to robot.

Finding: Joint-based (32-DOF) significantly outperforms the alternatives. The intuition is straightforward — if you want to teach a robot to "bend down, reach, grasp, and stand up," you need to capture exactly that motion. You can't reconstruct full-body coordination from three tracking points or decouple arm and leg intent.

Operators stand in a safe area wearing the PICO4U VR kit (HMD + controllers + leg trackers), and the G1 mirrors their motion in real time. The collected action space is 32-DOF, covering legs, torso, arms, and head pose.

Phase 2: VLA Policy Design for Whole-Body Control

With 32-DOF demonstration data collected, the next question is: which model to train?

OpenHLM uses π0.5 (pi05) from Physical Intelligence — a diffusion-based VLA pretrained on large amounts of static robot arm and wheeled dual-arm data. Notably, this model has never been trained on humanoid data, yet transfers well to G1 whole-body control after fine-tuning.

Two key findings:

1. Non-humanoid pretraining transfers to humanoid action spaces: The common assumption is that humanoid pretraining is necessary for humanoid fine-tuning. OpenHLM disproves this: π0.5 pretrained on static arms and wheeled platforms transfers effectively to a 34-DOF whole-body action space. Primitive skills — grasping, approaching a target, placing objects — appear to be transferable knowledge despite large embodiment gaps.

2. Multi-step inference outperforms one-step by ~20 task progress points: Diffusion models can run in one-step (single denoising pass, fast) or multi-step (iterative denoising, accurate) modes. For a 34-DOF action space with a 50-timestep horizon, multi-step inference is clearly better — the complex action space needs more denoising steps to "sculpt" precise motor commands.

Model configuration:

Phase 3: HuMI Co-Training — Scaling Data Without Robots

This is OpenHLM's most creative contribution. HuMI (Human Motion Imitation) collects demonstration data without using the robot at all:

1. A person performs the real task (picking up, placing objects...)
2. 5 HTC Vive trackers on the body capture full-body motion
3. 2 GoPro cameras on the wrists capture wrist-eye-view video (matching the robot's camera configuration)
4. IK software processes tracker signals → corresponding G1 joint angles
5. GoPro video is aligned with joint trajectories → ready-to-train demonstration dataset

The key insight: HuMI data costs far less than robot teleoperation. No robot time required, no VR headset, no specialized operator training. People simply perform tasks naturally while wearing trackers.

Three data types in co-training:

Results of HuMI co-training: 84–89% task progress on held-out tasks versus 36% baseline with full-body teleop alone. HuMI dramatically expands generalization to novel objects and language instructions the robot has never seen during teleoperation.

Installation

1. PC-side environment setup

# Clone the main repo
git clone https://github.com/OpenHLM-project/OpenHLM.git
cd OpenHLM

# Setup teleop environment
cd src/GR00T-WholeBodyControl4OpenHLM/
uv pip install --python .venv_teleop/bin/python -e ./gear_sonic/thirdparty/GMR
uv pip install --python .venv_teleop/bin/python -e ./openpi-client

# Build video sender tool
cd ./gear_sonic/thirdparty/XRoboToolkit-Orin-Video-Sender
sudo apt update && sudo apt install build-essential pkg-config libopencv-dev \
  libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libglib2.0-dev \
  libzmq3-dev libssl-dev
make

2. Robot-side setup

git clone https://github.com/Tendourisu/hardware4OpenHLM.git
cd hardware4OpenHLM
uv run hardware_setup

3. VLA training environment

cd src/openpi4OpenHLM
GIT_LFS_SKIP_SMUDGE=1 uv sync
GIT_LFS_SKIP_SMUDGE=1 uv pip install -e .

# Fix transformers compatibility
cp -r ./src/openpi/models_pytorch/transformers_replace/* \
  .venv/lib/python3.11/site-packages/transformers/

# Download π0.5 pretrained checkpoint
uv run examples/convert_jax_model_to_pytorch.py \
    --checkpoint_dir gs://openpi-assets/checkpoints/pi05_base \
    --config_name pi05_aloha \
    --output_path ~/.cache/openpi/openpi-assets/checkpoints/pi05_base_pytorch

Data Collection

Robot teleoperation

# Terminal 1: On the robot (G1 side)
cd path/to/hardware4OpenHLM
uv run hardware_setup

# Terminal 2: On the PC side
cd src/GR00T-WholeBodyControl4OpenHLM/scripts
bash launch_data_collection.sh

Safety note: Always start with reduced gains. The G1 weighs 35kg — incorrect motion commands can cause injury or hardware damage. Keep an emergency stop within reach at all times.

HuMI data pipeline

# Online IK with Vive tracker streaming
uv run online-ik pick --rpc-address tcp://<windows_machine_ip>:4242

# Align GoPro video and create dataset plan
uv run run-pipeline data/<humi_session> --gopro_timezone +08:00

# Export to OpenHLM format
uv run generate-final data/<humi_session>

HuMI hardware: 5 HTC Vive trackers, 1 Windows machine running SteamVR, 2 GoPro cameras (wrist-mounted), 1 Linux/macOS machine for IK processing.

Training Pipeline

Step 1: Convert data to LeRobot format

# Single dataset
uv run examples/unitree_g1/convert_g1_data_to_lerobot.py \
    --data_dir ./example_demonstration/20260520_1602_example_1 \
    --repo_name OpenHLM/example

# Multiple datasets
uv run examples/unitree_g1/convert_g1_data_to_lerobot_multi.py \
    --parent_dir ./example_demonstration \
    --dataset_folders 20260520_1602_example_1 20260520_1602_example_2 \
    --repo_name OpenHLM/example

Step 2: Compute normalization statistics

CUDA_VISIBLE_DEVICES=0 uv run scripts/compute_norm_stats.py \
    --config-name openhlm_example \
    --max-frames 500000

Step 3: Train

CUDA_VISIBLE_DEVICES=0,1,2,3 uv run torchrun --standalone --nnodes=1 \
  --nproc_per_node=4 scripts/train_pytorch.py \
  openhlm_example \
  --save-interval 10000 \
  --batch_size 128 \
  --no-enable-gradient-checkpointing \
  --enable-training-compile \
  --pytorch-weight-path ~/.cache/openpi/openpi-assets/checkpoints/pi05_base_pytorch \
  --exp-name openhlm_example

Training requires 4 GPUs (RTX 5080 class). 30,000 steps takes a few hours.

Inference and Deployment

Launch policy server

cd src/openpi4OpenHLM
CUDA_VISIBLE_DEVICES=0 uv run scripts/serve_policy.py \
  --env SONICG1 \
  --num-steps 10 \
  policy:checkpoint \
  --policy.config=openhlm_example \
  --policy.dir=path/to/trained/checkpoint

Run on robot

# Terminal 1: Low-level control server
cd src/GR00T-WholeBodyControl4OpenHLM/scripts
bash deploy_stream.sh

# Terminal 2: OpenPI deployment client
python openpi-eval/main.py \
  --control_hz 30 \
  --max_steps 10000 \
  --save_video \
  --instruction "pick up the apple and place it on the shelf" \
  --exp_name openhlm_example

The policy server runs on GPU, receives camera observations, and outputs 34-DOF action sequences at 30Hz. The low-level WBC controller receives actions and executes them on the robot.

Pre-trained checkpoints

If you don't want to train from scratch, download from HuggingFace:

Results: Beating GR00T N1.6 with Half the Demos

OpenHLM was evaluated on the HLM-12 benchmark — 12 language-conditioned tasks requiring full-body coordination spanning a wide vertical range (from floor level to above-shoulder height).

Comparison with baselines:

OpenHLM achieves 87.5% task progress on the fruit arrangement long-horizon task — arranging 5 types of fruit on shelves in a specified order — outperforming both state-of-the-art baselines using fewer than half their total demonstration time. Averaged across all 12 tasks, task progress exceeds 90%.

Key Engineering Takeaways

Three insights from OpenHLM worth remembering for anyone doing robot learning:

1. Action space design matters more than demo count: Joint-based 32-DOF teleoperation takes more effort to set up, but produces qualitatively better demonstrations than task-space or decoupled alternatives. With robot learning, data quality determines the ceiling — not the number of episodes.

2. Cross-embodiment transfer is larger than you think: π0.5 was trained on static arms and wheeled platforms, yet fine-tunes well onto a 34-DOF humanoid. Primitive skills — grasping, approaching, placing — appear to transfer across large embodiment gaps. This means you don't always need humanoid-specific pretraining data.

3. HuMI breaks the data collection bottleneck: Robot teleoperation is expensive: robot time, VR operators, space. HuMI collects from humans doing tasks naturally — scalable, cheap, and it meaningfully improves generalization. For any lab constrained by robot availability, this direction is worth investing in.

For a deeper look at whole-body control foundations, see the GR00T N1 architecture deep-dive. For context on how different VLA approaches handle whole-body tasks, the WholebodyVLA tutorial from OpenDriveLab covers a complementary approach. The Gear Sonic Humanoid WBC series also covers practical WBC deployment on humanoid platforms.

Conclusion

OpenHLM doesn't chase the largest model or the most impressive hardware. Instead, it answers a practical question: "If you have a Unitree G1 and want to teach it whole-body loco-manipulation, what should you do?"

The answer is a three-phase recipe: joint-based 32-DOF teleoperation, π0.5 VLA with multi-step inference, and HuMI co-training for scalable heterogeneous data. Together, these outperform both GR00T N1.6 and Ψ₀ while using less than half the demonstration time — and everything is open-source.

Paper: arXiv 2606.22174 | Code: GitHub | Data: HuggingFace

Related Posts

• GR00T N1 Architecture: The Foundation VLA for Unitree G1
• Gear Sonic Humanoid Whole-Body Control: From Sim to Real
• WholebodyVLA Tutorial: Train and Deploy Full-Body VLA