StarVLA-WBC: train G1 whole-body VLA

Why StarVLA-WBC matters

StarVLA-WBC is a focused branch of the StarVLA ecosystem for humanoid whole-body control. It connects egocentric images, language instructions, and robot state to long-horizon humanoid actions, while leaving balance, contact handling, joint limits, real-time tracking, and emergency behavior to an external WBC stack. The original project repository is starVLA/starVLA-WBC. The foundation paper is StarVLA: A Lego-like Codebase for Vision-Language-Action Model Developing, which presents StarVLA as a modular VLA research platform: visual-language or world-model backbones are separated from action heads, datasets are separated from model-specific preprocessing, and benchmarks share a common training/evaluation/deployment interface.

The most useful idea in StarVLA-WBC is not "replace whole-body control with a VLA." It is almost the opposite: do not make the VLA do the controller's job. The VLA should decide what the humanoid intends to do over the next action chunk: where the arms should move, what the hands should do, how the torso and base should progress, and which high-level motion intent is needed for the task. The WBC controller should still own high-frequency stability, proprioceptive history, contact constraints, PD targets, actuator limits, and safety. This split is especially important for Unitree G1 because large VLA models are usually much slower than the low-level loops that keep a biped upright.

If you have read our GR00T N1 G1 WBC deployment guide, the shape will feel familiar: a GPU-side policy server produces high-level actions, while the G1-side controller stack executes them. If ASAP for G1 is mainly about correcting sim-to-real dynamics mismatch, StarVLA-WBC is about a different engineering question: how do we standardize datasets, action contracts, and policy serving so that a whole-body VLA policy can plug into controller stacks such as SONIC, AMO, HOMIE, BFM-Zero, or a lab-specific G1 controller?

Paper and project idea for beginners

A common VLA policy for a robot arm receives an image, a task instruction, maybe a short proprioceptive state, and predicts a 6-7 DoF action: an end-effector delta, a gripper command, or a joint target. A humanoid like Unitree G1 is harder. It has legs, a waist, two arms, hands, a floating base, underactuated balance, and contact transitions. If the model directly predicts motor torques or low-level joint targets at high frequency, it must learn perception, language grounding, planning, locomotion stability, and actuator behavior in one model. That is hard to debug, sensitive to latency, and risky on hardware.

StarVLA-WBC uses a layered design:

camera + instruction + robot state
        -> StarVLA-WBC policy
        -> humanoid command / action chunk
        -> external WBC controller
        -> simulator or Unitree G1

The core StarVLA technical report contributes the "Lego-like" model organization. A researcher can swap a Qwen-VL backbone for a world-model backbone, swap an OFT action head for a flow-matching Pi-style head, or move from LIBERO to SIMPLE or RoboCasa without rewriting the whole stack. StarVLA-WBC keeps that philosophy but narrows the surface to humanoids. It keeps model families such as StarVLA-OFT, StarVLA-PI, StarVLA-GR00T, and WM4A/WanPI, while adding humanoid-specific pieces: HumanoidCommand, action codecs, controller adapters, and real-robot G1 workflow scaffolding.

For a beginner, the stack is easiest to remember in three layers:

The word "contract" is doing serious work here. If the dataset stores a 78D vector, you must know that the first 64 dimensions are a SONIC motion token, the next 7 are left hand joints, and the final 7 are right hand joints. If the dataset stores a 36D simple_psi0 vector, you must know that it includes hands, arms, torso, base height, planar velocity, yaw rate, and target yaw. If you change joint order but leave the config untouched, the model may still run, but the robot will do the wrong thing. In humanoid VLA, the most dangerous bugs are often semantic mismatches, not Python exceptions.

StarVLA-WBC architecture

StarVLA-WBC separates the humanoid interface into four contracts. First, an action codec decodes a flat model output into a semantic command. Second, HumanoidCommand carries named joint positions/velocities and floating-base motion fields. Third, a controller adapter maps that command into the native input format of one controller runtime. Fourth, a ControllerBackend returns a controller-neutral HumanoidJointTarget to the simulator or robot bridge.

In the SIMPLE integration, one sample contains one egocentric RGB image, one language instruction, and a 32D state. The older default policy predicts a 30-step action chunk, each step using a 36D layout:

This 36D layout does not contain leg joints. It is an upper-body joint reference plus floating-base motion intent, not a full motor command. That is why one frozen policy can be evaluated with several backends. SONIC synthesizes lower-body motion and balance, AMO owns locomotion and pelvis behavior, HOMIE builds its official 456D observation history for a leg ONNX policy, and BFM-Zero uses motion completion plus its own encoder/actor path. The repo also supports a newer g1_joint_reference 48D layout: 29 body joints, 14 hand joints, and five root-motion fields for planar velocity, yaw rate, base height, and target yaw. This is closer to a controller-neutral full-body reference, but it only helps if the backend can actually track full-body trajectories.

For the real Unitree G1 route, StarVLA-WBC documents a SONIC/Dex3 example with state 72D, action 78D, horizon 8, and q99 normalization. The 78D action means:

action[0:64]   -> SONIC motion latent
action[64:71]  -> left hand joints
action[71:78]  -> right hand joints

This route is practical if you already use GR00T-WholeBodyControl or a SONIC-style WBC stack, because high-frequency balance stays in a controller designed for G1. The tradeoff is that a 64D motion latent is less controller-neutral than a semantic 36D or 48D reference. It is tied to the decoder and controller family used to collect and execute the demonstrations.

Installation

Use a Linux workstation with an NVIDIA GPU, CUDA-enabled PyTorch, Python 3.10 or newer, and enough disk for datasets and checkpoints. SIMPLE evaluation runs in a separate SIMPLE/Isaac environment and talks to the policy server over WebSocket. Real G1 deployment also needs Unitree SDK setup, camera servers, network configuration, and a WBC controller stack, but those pieces should not be mixed into the VLA training environment.

Install StarVLA-WBC:

git clone https://github.com/starVLA/starVLA-WBC.git
cd starVLA-WBC
python -m pip install -r requirements.txt
python -m pip install -e .

Place or symlink a supported Qwen checkpoint under:

playground/Pretrained_models/

Alternatively, override framework.qwenvl.base_vlm with a Hugging Face model ID or a local path. The SIMPLE recipe uses Qwen3-VL-4B-Instruct by default. The real G1 SONIC smoke route uses a smaller Qwen3.5-0.8B path in the example YAML. If your GPU budget is limited, start with an OFT/MLP smoke test, batch size 1, and MAX_STEPS=1000 before attempting a full training run.

A practical setup checklist:

1. python -c "import torch; print(torch.cuda.is_available())" returns True.
2. python -m pip show accelerate transformers torch points to the intended environment.
3. The dataset root is on fast local or shared storage.
4. The Qwen checkpoint loads in a fresh process.
5. DRY_RUN=1 prints the resolved command before real training starts.

Training on SIMPLE

SIMPLE is a Unitree G1 whole-body simulation benchmark. StarVLA-WBC integrates it for tasks such as XMovePick, BendPick, Handover, LocoPickBetweenTables, TabletopGrasp, CloseDoor, OpenOven, OpenFaucet, and PickAndPlaceAndHugContainer. Training uses LeRobot v2.1 datasets; evaluation runs in SIMPLE/Isaac and calls the policy server at 50 Hz.

Download the 10-task training data:

export SIMPLE_TASKS="G1WholebodyXMovePickTeleop-v0,G1WholebodyBendPickMP-v0,G1WholebodyHandoverTeleop-v0,G1WholebodyLocomotionPickBetweenTablesTeleop-v0,G1WholebodyTabletopGraspMP-v0,G1WholebodyXMoveBendPickTeleop-v0,G1WholebodyCloseDoorTeleop-v0,G1WholebodyOpenOvenTeleop-v0,G1WholebodyOpenFaucetTeleop-v0,G1WholebodyPickAndPlaceAndHugContainerTeleop-v0"
bash examples/simBenchmarks/SIMPLE/train_files/download_simple_data.sh playground/Datasets

Train the default StarVLA-PI policy:

data_mix=simple_psi0_all_ten_tasks \
NUM_PROCESSES=8 \
bash examples/simBenchmarks/SIMPLE/train_files/run_simple_train.sh

For a simpler direct-regression baseline, run StarVLA-OFT with the OFT YAML:

config_yaml=examples/simBenchmarks/SIMPLE/train_files/starvla_simple_psi0_qwenoft.yaml \
Framework_name=StarVLA-OFT \
bash examples/simBenchmarks/SIMPLE/train_files/run_simple_train.sh

Useful variables include NUM_MACHINES, MACHINE_RANK, MAIN_PROCESS_IP, MAIN_PROCESS_PORT, MAX_TRAIN_STEPS, REPEATED_DIFFUSION_STEPS, and WANDB_MODE. If you are new to this stack, run a dry pass first:

DRY_RUN=1 \
data_mix=simple_psi0_all_ten_tasks \
bash examples/simBenchmarks/SIMPLE/train_files/run_simple_train.sh

The dry run does not train the model. It shows whether paths, process counts, model names, and overrides resolve correctly. Most first failures are mundane: wrong data_root_dir, missing checkpoint, too many GPU processes for the machine, or a stale base_vlm path. Catch those before launching a long job.

Training a real G1 route from LeRobot data

SIMPLE teaches the action contract in simulation. To move toward a physical Unitree G1, you need demonstration data with a stable schema. StarVLA-WBC documents a concrete reference path using NVlabs/GR00T-WholeBodyControl: collect teleop data, export LeRobot v2.1, train StarVLA, run a StarVLA WebSocket policy server, and keep SONIC/WBC on the controller side. The LeRobot Unitree G1 docs are also relevant because they now cover 23 DoF and 29 DoF G1 variants, simulation teleoperation, physical robot connection, recording, training, and real-time chunking inference.

The example dataset layout in StarVLA-WBC is:

playground/Datasets/UnitreeG1_WholeBody/lerobot/test_sonic/
  data/chunk-000/episode_000000.parquet
  videos/chunk-000/observation.images.ego_view/episode_000000.mp4
  meta/
    info.json
    modality.json
    episodes.jsonl
    tasks.jsonl

The documented schema reports fps 50, 49 episodes, 56,919 frames, video key observation.images.ego_view, and the task text "pick the toy on the table, and put it into the box." Key tensor dimensions include observation.state 43D, observation.eef_state 14D, action.wbc 43D, action.motion_token 64D, 7D left/right hand joints, teleop.smpl_pose 63D, 9D VR position, and 18D VR orientation. The recommended first action registry is the 78D controller target: action.motion_token plus teleop.left_hand_joints plus teleop.right_hand_joints.

Validate the dataset before training:

export DATASET=playground/Datasets/UnitreeG1_WholeBody/lerobot/test_sonic

python3 - <<'PY'
import json, os
from pathlib import Path
dataset = Path(os.environ["DATASET"])
info = json.loads((dataset / "meta/info.json").read_text())
modality = json.loads((dataset / "meta/modality.json").read_text())
print("fps:", info.get("fps"))
print("episodes:", info.get("total_episodes"))
print("features:", sorted(info.get("features", {}).keys()))
print("state groups:", sorted(modality.get("state", {}).keys()))
print("action groups:", sorted(modality.get("action", {}).keys()))
PY

Run the single-GPU smoke training path:

WANDB_MODE=disabled \
BATCH=1 \
MAX_STEPS=1000 \
bash examples/realRobots/UnitreeG1_WholeBody/step2_training/train_files/run_starvla_qwenoft_g1_sonic_train.sh

The example config starvla_qwenoft_g1_sonic.yaml sets framework StarVLA-OFT, an MLP action model, action_dim: 78, state_dim: 72, action_horizon: 8, image size 224x224, gradient accumulation 4, base learning rate 2e-5, Qwen interface learning rate 1e-5, action model learning rate 1e-4, save interval 200, and save_format: pt. With a small dataset, the first goal is not SOTA. The goal is to prove that the registry loads, one batch has shape [8, 78], model outputs are finite, a checkpoint can reload, and dataset_statistics.json stays with the checkpoint.

Inference and evaluation

For SIMPLE evaluation, run two processes: the policy server and the SIMPLE client. First download evaluation assets:

SIMPLE_ROOT=/path/to/SIMPLE \
SIMPLE_TASKS="G1WholebodyXMovePickTeleop-v0" \
bash examples/simBenchmarks/SIMPLE/eval_files/download_simple_eval_data.sh

Start the policy server:

bash examples/simBenchmarks/SIMPLE/eval_files/run_policy_server.sh \
  /path/to/checkpoints/steps_100000_pytorch_model.pt 0 22520

Run one task and one randomization level:

SIMPLE_ROOT=/path/to/SIMPLE \
STARVLA_SIMPLE_SERVER_HOST=127.0.0.1 \
STARVLA_SIMPLE_SERVER_PORT=22520 \
NUM_EPISODES=10 \
bash examples/simBenchmarks/SIMPLE/eval_files/run_simple_eval.sh \
  G1WholebodyXMovePickTeleop-v0 level-0

Choose a backend with STARVLA_SIMPLE_CONTROLLER=sonic, amo, homie, bfm-zero, or auto. auto keeps the native SIMPLE assignment: SONIC for Teleop tasks and AMO for MP tasks. When comparing controllers, keep the task, episode, randomization level, reset state, policy checkpoint, action normalization, action layout, execution horizon, language instruction, image preprocessing, 32D state, success criterion, and maximum episode steps fixed. StarVLA-WBC also calls out timing explicitly: the VLA backend is queried at 50 Hz; Teleop/SONIC holds the target across four 5 ms physics substeps, while the native G1 environment holds it across ten 2 ms substeps. Adding extra controller-side decimation, such as turning a 50 Hz backend into 12.5 Hz calls, invalidates the comparison.

Reported results

On the 10-task SIMPLE suite, StarVLA-WBC reports a single StarVLA-PI / QwenPi model co-trained for 100k steps with 245 successful episodes out of 300, or 81.67%. In the same aggregate table, the official SIMPLE Psi0 number uses a separate policy per task and reaches 226/300, or 75.33%. StarVLA-WanPI / WanPi, co-trained on 12 tasks and evaluated on the same 10-task subset, reaches 90/300, or 30.00%. The lesson is useful: a multi-task QwenPi policy can compete with per-task training, but not every backbone/action-head pairing is automatically strong.

The controller-backend alignment study freezes the same QwenPi step-100k checkpoint and keeps the model-facing request/action contract fixed. Only the backend changes. Native SIMPLE assignment reaches 245/300. HOMIE reaches 138/300, AMO 55/300, and BFM-Zero 62/300. Do not read this as an absolute controller ranking. It measures how well the frozen policy's learned 36D command distribution aligns with each adapter/runtime. If each backend had its own retrained policy, the numbers could change.

The practical engineering conclusion is simple: the best route is usually the one where the data collection controller, training action contract, and deployment controller match. If Teleop data was generated through SONIC, a SONIC deployment path has less semantic gap. If you switch backends, audit command semantics, controller observations, history buffers, native command order, control law, timing, and success criteria.

Deploying through WBC on Unitree G1

StarVLA-WBC does not ask StarVLA to own the entire Unitree stack. The recommended real-deployment shape is:

StarVLA checkpoint
  -> StarVLA WebSocket policy server
  -> G1 policy client / adapter
  -> user-owned controller stack
  -> Unitree G1

Start the server on the GPU machine:

export PYTHONPATH=$(pwd):${PYTHONPATH}

CUDA_VISIBLE_DEVICES=0 python deployment/model_server/server_policy.py \
  --ckpt_path /path/to/g1/checkpoints/steps_<N>_pytorch_model.pt \
  --port 5694 \
  --use_bf16

The G1 adapter reads camera images and robot state, then sends a StarVLA request:

{
    "examples": [{
        "image": [cam0, cam1],
        "state": state,
        "lang": prompt,
    }],
    "unnorm_key": "unitree_g1_sonic",
}

The server returns actions with shape [B, T, action_dim]. The adapter splits the flat action into controller groups, clips each group, checks stale actions, and publishes to the SONIC decoder, the C++ deploy side, or a custom G1 controller bridge. The adapter should not reimplement normalization math if the policy server already returns an unnormalized action chunk. Its job is grouping, clipping, freshness checks, and controller publishing.

A safe dry-run order is:

1. Replay a recorded episode through the policy and adapter.
2. Inspect every returned action group with actuators disabled.
3. Run the complete interface in simulation or controller dry-run.
4. Connect to the robot with the policy paused and verify state freshness.
5. Enable low-speed motion for a small set of action groups.
6. Gradually enable more groups after timing, limits, and emergency behavior are validated.

Minimum safety requirements include an emergency stop independent from StarVLA, per-group action clipping, velocity and acceleration limits, stale-action timeout, camera/state freshness checks, a policy pause command, controller health checks, and a clear operator procedure. If any item is missing, stay in simulation.

Common failure modes

The first common issue is camera mismatch. Training may use observation.images.ego_view, while deployment sends a global camera, a different crop, BGR instead of RGB, or a different resize pipeline. Save the first request audit: image, state, instruction, and predicted action chunk.

The second issue is state/action order mismatch. On G1, swapping left/right arms or flipping a hand command sign can create dangerous motion while every tensor still has the expected shape. Print meta/modality.json, compare it with data_config.py, and compare both with the adapter split logic.

The third issue is normalization mismatch. Older SIMPLE paths use min_max; the G1 SONIC route uses q99 through q01/q99 statistics. The checkpoint, dataset_statistics.json, and YAML config must travel together.

The fourth issue is timing mismatch. The action horizon may be 8 or 30; controller loops may run at 50 Hz, 200 Hz, or 500 Hz. You need to know how long each chunk is held, whether interpolation is applied, and when stale actions are discarded.

The final issue is misreading benchmarks. One model co-trained on 10 tasks is not the same as 10 task-specific models. A replacement-backend study is not the same as retraining for each backend. Always record training scope, checkpoint step, tasks, levels, episode count, action layout, and controller assignment.

Related Posts

• GR00T N1 G1 WBC deployment
• ASAP for agile Unitree G1 skills
• FineVLA dual-arm VLA tutorial