CoorDex: Training Unitree G1 to Walk and Dexterously Grasp Simultaneously with Body and Hand Latent Priors

Here is a problem that has frustrated robotics researchers for years: humanoid robots can walk well, or they can grasp objects well — but rarely both at the same time. Most systems follow a "stop-then-manipulate" paradigm where the robot walks to the target, halts completely, performs the manipulation, then resumes locomotion. This stop-and-go behavior is a major bottleneck for real-world deployment.

CoorDex (arXiv 2606.23680) from UNC Chapel Hill and UC Berkeley attacks this problem with an elegant insight: instead of learning directly in the 29-dimensional body joint space plus 20 finger joints, compress both whole-body and dexterous hand control into small structured latent spaces, then learn to coordinate them through a shared residual policy.

The result: Unitree G1 can walk toward a water bottle, reach and grasp it mid-stride, and carry it away — all continuously without stopping. This is the first demonstration of this capability on real hardware with a 7-DoF dexterous hand under continuous loco-manipulation.

Why Is This Problem So Hard?

Think about walking and picking something off a table. Your brain coordinates extraordinary complexity: legs maintain balance and generate steps, hips shift your center of mass, shoulder and arm approach the target, fingers precisely align for the grasp. All simultaneously and continuously.

For robots, the challenge multiplies:

1. Enormous action space — Unitree G1 has 29 actuated body joints plus 12-20 finger joints, totaling 41-49 action dimensions. Direct RL in this space struggles badly to converge because the exploration space is simply too vast.

2. Different timescales — Locomotion operates on step cycles of roughly 0.5-1 second, while finger manipulation needs much faster feedback for contact force handling.

3. Mechanical cross-coupling — Every footfall creates perturbations that propagate through the entire body, directly affecting wrist and finger positions. Finger control during walking must compensate for these oscillations in real time.

4. Sparse rewards — What exactly counts as "successfully grasped while walking"? Stopping to grasp does not count. Designing reward functions for continuous loco-manipulation is highly non-trivial.

Prior approaches — direct full-joint PPO learning, or separating locomotion and manipulation into two modules — all fail completely under the same reward budget, as confirmed by CoorDex's ablation study.

CoorDex Architecture: Three Sequential Stages

CoorDex trains in three sequential phases, each addressing a different level of complexity:

Stage 1: Privileged Motion-Tracking Teachers

Body Teacher learns locomotion and reaching from reference motions:

• Input: body proprioception (joint positions/velocities, base angular velocity, 5-frame history) + reference goals
• Output: 29 body joint position targets, excluding fingers
• Training: RL with reference tracking rewards + domain randomization (floor friction, torso COM perturbation)
• Key design: privileged critic — the critic sees reference motion errors that the actor cannot, yielding a more accurate value function while keeping the deployment policy robust

Hand Teacher learns dexterous grasping in a separate floating-hand environment:

• Dataset: ManipTrans-style retargeted hand-object motions derived from MANO keypoints of human demonstrations
• Critical trick: wrist-stabilized configuration — wrist pose is written directly into the simulator from reference data; the hand teacher only needs to control fingers
• Why this matters: if the hand teacher had to also learn wrist motion, the 12-dimensional latent space would be consumed by 6D wrist kinematics instead of capturing the finger dexterity that makes grasping reliable

Stage 2: VAE Latent Prior Distillation

Each teacher is distilled into a proprioception-conditioned latent prior via a Variational Autoencoder:

Encoder:  full_observation + reference_goals → latent distribution q(z|obs, goal)
Prior:    proprioception_only → latent distribution p(z|proprio)  ← used at deployment
Decoder:  proprioception + z → joint actions  ← FROZEN after training

• Body latent space: 16-dimensional → decodes to 29 body joint targets
• Hand latent space: 12-dimensional → decodes to 12 finger targets (Dex3-1 real hand) or 20 targets (WUJI simulation hand)

Why VAE instead of plain behavioral cloning? VAE creates a smooth, structured latent space — nearby points correspond to similar actions in joint space. This is essential because the residual policy in stage 3 learns to navigate this latent space meaningfully.

Total action dimensionality through the latent interface: 16 (body) + 12 (hand) = 28 dimensions, down from 41-49 in raw joint space — but more importantly, the space is structured in a way that makes coordination learnable.

Stage 3: Coordinated Latent Residual Policy (PPO)

This is the "coordination brain" — a policy that learns to slightly adjust (not replace) both priors simultaneously through a shared task context:

Input:  [proprioception, task_state, object_geometry,
         contact_features, μ^b,p (body prior mean), μ^h,p (hand prior mean)]
                         ↓
        Shared Coordination Trunk (MLP [512, 256])
                         ↓
            ┌────────────────────────┐
            ↓                        ↓
       Body Residual Head        Hand Residual Head
           (Δz^b)                    (Δz^h)
            ↓                        ↓
  z̃^b = μ^b,p + Δz^b      z̃^h = μ^h,p + Δz^h
            ↓                        ↓
      Body Decoder             Hand Decoder (FROZEN)
            ↓                        ↓
      Body joint targets       Finger joint targets

Core formula: z̃ = μ_prior + Δz_residual

When Δz is small, the prior behaves as originally learned — natural locomotion patterns are preserved. The policy only pushes large residuals when necessary, such as adjusting wrist position to approach an object or modulating finger force during contact.

This is analogous to expert musicians: technical fundamentals live in muscle memory (latent prior), while conscious attention focuses on small real-time adjustments for the specific piece (residual policy).

Hardware and Simulation Setup

Unitree G1:

• Height: ~127 cm, weight: ~35 kg
• 29 actuated body joints
• Real hand: Dex3-1 from Inspire Robots (7-DoF)
• Simulation hand: WUJI (20-DoF, 5 fingers × 4 joints each)
• Total latent dimensions: 16 body + 12 hand = 28

Isaac Lab (GPU-accelerated physics):

• 4,096 parallel environments on RTX 4090
• Throughput: 1 second of simulation ≈ 27 minutes of real-world experience
• Physics: NVIDIA PhysX 5

Three trained tasks:

Installation: Step by Step

Minimum System Requirements

GPU:   NVIDIA RTX 4090 (24 GB VRAM) — minimum for training
       NVIDIA A100 80 GB — recommended for faster training
RAM:   32 GB or more
OS:    Ubuntu 20.04 or 22.04
CUDA:  12.x

Environment Setup

# Step 1: Clone the CoorDex repository
git clone https://github.com/Skevinci/CoorDex
cd CoorDex

# Step 2: Install NVIDIA Isaac Sim (follow official documentation)
# See: https://isaac-sim.github.io/IsaacLab/
# Requires: NVIDIA Isaac Sim 4.x or later

# Step 3: Create a Python conda environment
conda create -n coordex python=3.10
conda activate coordex

# Step 4: Install Python dependencies
pip install -r requirements.txt

# Step 5: Install Isaac Lab extensions
python -m pip install -e source/extensions/omni.isaac.lab
python -m pip install -e source/extensions/omni.isaac.lab_tasks

# Step 6: Verify the installation
python scripts/check_env.py

Repository Structure

CoorDex/
├── envs/                    # Isaac Lab task environments
│   ├── walk_grab.py         # WalkGrab task definition
│   ├── open_fridge.py       # OpenFridge task definition
│   └── walk_pick_turn.py    # WalkPickTurn task definition
├── models/                  # Model architectures
│   ├── body_prior/          # VAE body latent prior
│   ├── hand_prior/          # VAE hand latent prior
│   └── coord_policy/        # Coordinated residual policy
├── scripts/                 # Training and evaluation
│   ├── train_body_teacher.py
│   ├── train_hand_teacher.py
│   ├── distill_prior.py
│   ├── train_coord_policy.py
│   └── deploy_real.py       # Real robot inference
├── configs/                 # YAML hyperparameter configs
└── data/
    └── mano_retargeted/     # ManipTrans-processed hand motion data

Training Pipeline: Practical Walkthrough

Step 1: Train the Body Teacher

python scripts/train_body_teacher.py \
  --robot unitree_g1 \
  --num_envs 4096 \
  --max_iterations 5000 \
  --log_dir logs/body_teacher

The privileged critic design is important here: during training the critic receives reference motion error signals that the actor cannot access. This allows the critic to learn an accurate value function for guiding policy updates, while the actor remains constrained to observations available at deployment (proprioception only).

Domain randomization applied during training:

• Floor friction coefficient: ±30% of nominal
• Random external force perturbations on the torso
• Joint angle observation noise: ±0.01 rad

Step 2: Train the Hand Teacher (Wrist-Stabilized)

python scripts/train_hand_teacher.py \
  --mode wrist_stabilized \
  --dataset data/mano_retargeted/ \
  --num_envs 4096 \
  --max_iterations 3000 \
  --log_dir logs/hand_teacher

The most important configuration: wrist_pose_override: True. The simulator reads wrist pose directly from the reference trajectory and sets it in the simulation state each step. The hand teacher only needs to solve the finger control problem.

Hand teacher reward components:

• MANO keypoint tracking loss (fingertips weighted more heavily)
• Velocity tracking for finger joints
• Power penalty to discourage energy waste
• Action rate penalty to encourage smooth motion

Step 3: Distill VAE Latent Priors

# Distill body latent prior
python scripts/distill_prior.py \
  --type body \
  --teacher_ckpt logs/body_teacher/best.pt \
  --latent_dim 16 \
  --output models/body_prior/ \
  --epochs 200

# Distill hand latent prior
python scripts/distill_prior.py \
  --type hand \
  --teacher_ckpt logs/hand_teacher/best.pt \
  --latent_dim 12 \
  --output models/hand_prior/ \
  --epochs 200

VAE training uses a combined loss function:

L_total = L_reconstruction + λ_KL × KL(q||p) + λ_smooth × temporal_smoothness

Where:
  L_reconstruction = MSE between predicted and teacher actions
  KL(q||p)         = KL divergence between encoder and prior distributions
  temporal_smooth  = mean ||z_t - z_{t-1}||² across the trajectory

The temporal smoothness term is non-optional — without it, latent trajectories jump erratically between frames, causing unacceptable jitter at the joint output level.

Step 4: Train the Coordinated Residual Policy

python scripts/train_coord_policy.py \
  --task walk_grab \
  --body_prior models/body_prior/ \
  --hand_prior models/hand_prior/ \
  --num_envs 4096 \
  --rollout_steps 24 \
  --num_minibatches 4 \
  --learning_rate 1e-3 \
  --gamma 0.99 \
  --gae_lambda 0.95 \
  --kl_target 0.01 \
  --log_dir logs/coord_policy/walk_grab/

Key PPO hyperparameters:

• 4 minibatches × 98,304 samples = ~393K transitions per gradient update
• Adaptive learning rate scaled to maintain KL target of 0.01
• Actor/Critic MLP: [1024, 512, 256] with ELU activations
• Coordination trunk: [512, 256] shared → [256, 128] per residual head
• Discount factor γ = 0.99, GAE λ = 0.95

Both the body and hand decoders are frozen during this stage. Only the coordination trunk and two residual heads receive gradient updates. This preserves the motion priors learned in earlier stages.

Results

Task Success Rates (Simulation and Real World)

Ablation: Why Coordinated Structure Matters

The ablation reveals the key insight: it is not latent spaces in general that matter, but the separate dual-head coordination structure. A monolithic single latent space that combines body and hand also fails completely. Additionally, CoorDex achieves a lower action rate (0.22 vs 0.40), indicating smoother and less jerky control output.

Real-World Demo on Unitree G1

Deployment on Real Unitree G1

Running the Deployment Script

# Connect to Unitree G1 via network (typically 192.168.123.x subnet)
# Ensure the robot is in low-level control mode

python scripts/deploy_real.py \
  --task walk_grab \
  --policy logs/coord_policy/walk_grab/best.pt \
  --body_prior models/body_prior/ \
  --hand_prior models/hand_prior/ \
  --hand dex3_1

Handling the Sim-to-Real Gap

An interesting aspect of CoorDex: it trains with the WUJI 20-DoF hand in simulation but deploys with the 7-DoF Dex3-1 in the real world. The framework handles this morphology gap as follows:

• The hand decoder is retrained for Dex3-1's 7-DoF configuration
• The body prior and coordination trunk transfer directly — no fine-tuning needed
• Only the hand prior requires fine-tuning for the different output dimensionality

Domain randomization during training covers:

• Floor friction: ±30% of nominal
• Random external force perturbations on the torso
• Actuator delay: 10-30 ms
• Observation delay: 0-20 ms

Comparison with Other Approaches

CoorDex is closest in spirit to ω-0 (WAM) — both use latent predictive structure to coordinate locomotion and manipulation. The key difference: CoorDex focuses on dexterous finger-level grasping with a real multi-finger hand, while ω-0 addresses whole-body coordination at a higher level.

Compared to ASAP — also trained on Unitree G1 — CoorDex tackles a harder problem by coordinating locomotion and manipulation simultaneously rather than optimizing sim-to-real transfer of a single behavior.

Current Limitations and Future Directions

What CoorDex cannot yet do:

1. Limited object diversity: Currently trained on specific objects (water bottles, boxes). New objects require retraining or fine-tuning. Integration with a vision foundation model could extend generalization.

2. 55% WalkGrab is not production-ready: The paper traces most failures to the initial contact phase when the robot must simultaneously step and reach. Adding force sensing like FM-VLA could improve this phase.

3. Single-arm only: Extension to bimanual manipulation would require a stronger body prior to maintain balance with both arms occupied.

4. No semantic understanding: The robot does not "know" what it is grasping. Combining CoorDex with a high-level VLA policy could add semantic flexibility.

Why CoorDex Is a Meaningful Step Forward

The latent prior coordination approach represents a paradigm shift in humanoid controller design:

• Before: Learn directly in 40+ dimensional joint space, or separate locomotion and manipulation into independent modules
• CoorDex: Compress knowledge into structured latent spaces → coordinate via small residuals → preserve natural motion primitives

Think of it like learning basketball: once you have dribbling internalized as muscle memory (latent prior), you can focus on reading defenders and deciding where to move — simultaneously, without thinking about each wrist flick. CoorDex gives robots a similar ability to operate at multiple levels of abstraction simultaneously.

With open-source code available at github.com/Skevinci/CoorDex, this is an important baseline for anyone working on continuous dexterous loco-manipulation on humanoid platforms.

Related Posts

• ω-0: Latent Predictive World Action Model for Humanoid Loco-Manipulation
• ASAP: Training Unitree G1 with Delta Action Sim-to-Real Transfer
• FM-VLA: Force-Based Memory for Contact-Rich Manipulation