unifolm-vla + Unitree G1 (Post 1): WBC+VLA system architecture — from data to real hardware

This is the first post of a 5-part series on building a Whole-Body Control + Vision-Language-Action (WBC+VLA) system for the Unitree G1 humanoid robot — from real hardware data collection to deploying on the real robot.

This series differs from the GR00T N1 + G1 series: instead of NVIDIA GR00T-WBC (which requires special access and complex integration), we use the Unitree native stack — tools developed by Unitree themselves that work out of the box on G1 without external SDKs.

The problem: why do humanoids need WBC+VLA?

Humanoid robots face two simultaneous challenges:

Problem 1 (Locomotion):        Problem 2 (Manipulation):
G1 must stay balanced          G1 must see an object
while standing / walking       and control arms + fingers
→ 12 DOF (legs)               → 7+7+2+2 DOF (arms)
→ Runs at 200–500 Hz           → Runs at 3–10 Hz (VLM inference)

These two control loops can't share the same controller — the frequencies are too different. The solution is decoupled WBC+VLA:

┌─────────────────────────────────────────────┐
│              G1 Whole-Body                  │
├──────────────────┬──────────────────────────┤
│   ARM VLA        │      LOCOMOTION           │
│  unifolm-vla     │   G1 built-in SDK         │
│  Qwen2.5-VL-7B   │   or unitree_rl_gym       │
│  ~3-5 Hz         │   200-500 Hz              │
│  Controls arms   │   Controls legs           │
└──────────────────┴──────────────────────────┘
             ↕                  ↕
        xr_teleoperate      motion.pt
       (arm data)          (pretrained loco)

Both systems run in parallel — G1 walks while its arms perform tasks.

Three core repos

1. unifolm-vla — VLA for G1 arms

GitHub: https://github.com/unitreerobotics/unifolm-vla

unifolm-vla is a Vision-Language-Action model developed by Unitree for their humanoid robots, particularly G1:

• Backbone: Qwen/Qwen2.5-VL-7B-Instruct — a 7B parameter VLM by Alibaba, publicly available on HuggingFace
• Action head: predicts joint commands for G1 arms (7+7 DOF arm + 2+2 DOF gripper)
• Base checkpoint: Unifolm-VLM-0 — Unitree's continued-pretrained checkpoint, not yet public as of this writing
• Workaround: in this series, we fine-tune directly from Qwen2.5-VL-7B-Instruct (public) — performance is somewhat lower but entirely feasible

# Key repo structure
unifolm-vla/
├── src/unifolm_vla/
│   ├── training/train_unifolm_vla.py    # training script
│   ├── config/deepseeds/               # DeepSpeed configs
│   └── models/                         # model definitions
├── prepare_data/
│   ├── convert_lerobot_to_hdf5.py      # data pipeline step 2
│   └── hdf5_to_rlds/                   # data pipeline step 3
└── run_real_eval_server.py             # FastAPI inference server

2. xr_teleoperate — VR data collection

GitHub: https://github.com/unitreerobotics/xr_teleoperate

This tool collects teleoperation data via XR headsets:

Data output: JSON format containing time series of hand poses + joint states + camera frames.

3. unitree_rl_gym — G1 locomotion

GitHub: https://github.com/unitreerobotics/unitree_rl_gym

RL training for G1 locomotion using Isaac Gym (NOT IsaacLab). Key point:

• Includes motion.pt — pretrained G1 locomotion policy (public)
• For basic stand/walk/run behavior, no retraining needed — use motion.pt directly
• Retrain only for custom locomotion (e.g., stair climbing, faster running)

Hardware requirements

G1 hardware needed

Workstation requirements

Important: RTX 4090 is the minimum for training. For deploy only (inference server), an RTX 3080 10GB is sufficient — Qwen2.5-VL-7B can run at 4-bit quantization with ~8GB VRAM.

Environment setup

Step 1: Clone repos

mkdir ~/unifolm_ws && cd ~/unifolm_ws

# Repo 1: unifolm-vla (VLA model)
git clone https://github.com/unitreerobotics/unifolm-vla.git

# Repo 2: xr_teleoperate (data collection)
git clone https://github.com/unitreerobotics/xr_teleoperate.git

# Repo 3: unitree_rl_gym (locomotion)
git clone https://github.com/unitreerobotics/unitree_rl_gym.git

# Auxiliary: unitree_IL_lerobot (data conversion)
git clone https://github.com/unitreerobotics/unitree_IL_lerobot.git

Step 2: Create conda environments

Each repo needs its own environment (dependency conflicts):

# Environment for xr_teleoperate
conda create -n xr_teleop python=3.10
conda activate xr_teleop
cd xr_teleoperate
pip install -r requirements.txt

# Environment for unifolm-vla (training + inference)
conda create -n unifolm python=3.10
conda activate unifolm
cd unifolm-vla
pip install -e .
pip install deepspeed accelerate

# Environment for unitree_rl_gym (Isaac Gym-based, needs special setup)
conda create -n loco python=3.8
conda activate loco
cd unitree_rl_gym
pip install -e .

Step 3: Install Unitree SDK Python

Required for the deploy step:

conda activate unifolm
pip install unitree_sdk2py

Verify G1 connection:

python -c "
from unitree_sdk2py.core.channel import ChannelFactory
factory = ChannelFactory.Instance()
factory.Init(0, 'eth0')   # replace eth0 with your network interface
print('G1 connected OK')
"

Step 4: Download Qwen2.5-VL-7B-Instruct

conda activate unifolm
pip install huggingface_hub

python -c "
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id='Qwen/Qwen2.5-VL-7B-Instruct',
    local_dir='~/models/Qwen2.5-VL-7B-Instruct',
    ignore_patterns=['*.gguf', '*.bin']  # safetensors only
)
print('Download complete!')
"

Download is ~15GB — need ~30GB free disk (model + optimizer states).

End-to-end data flow

The full series at a glance:

[Post 2] Data collection
xr_teleoperate → teleop_hand_and_arm.py → JSON files
        ↓
[Post 3] Data pipeline
JSON → unitree_IL_lerobot → LeRobot V2.1 format
     → unifolm-vla prepare_data → HDF5
     → hdf5_to_rlds → RLDS format
        ↓
[Post 4] Training
RLDS → accelerate launch train_unifolm_vla.py → checkpoint
        ↓
[Post 5] Deploy
checkpoint → run_real_eval_server.py (FastAPI /act)
           → SSH tunnel → G1 robot client
           + motion.pt → G1 locomotion (parallel)

Post 1 summary

You now understand:

• unifolm-vla uses Qwen2.5-VL-7B backbone (VLA for G1 arms)
• xr_teleoperate collects data via VR headset (Meta Quest 3 recommended)
• unitree_rl_gym provides locomotion via pretrained motion.pt (no retraining needed)
• All three systems run in parallel for "whole-body control"
• Unifolm-VLM-0 not public → we fine-tune from Qwen2.5-VL-7B-Instruct (public)

Next post: detailed guide to collecting data with xr_teleoperate + Meta Quest 3 on real G1.

References

• unifolm-vla GitHub
• xr_teleoperate GitHub
• unitree_rl_gym GitHub
• Qwen2.5-VL-7B-Instruct (HuggingFace)

Related posts

• Post 2: Data collection with xr_teleoperate + Meta Quest 3
• VLA + WBC repos landscape 2025-2026
• GR00T N1 + G1: GEAR+SONIC+N1.5 architecture