Humanoid Adaptation: Extending VIMA to High-DoF Humanoid Robots

The previous four parts disassembled VIMA layer by layer: Part 1 on cross-attention architecture, Part 2 on 17 tasks and 4 generalization levels, Part 3 on the Mask R-CNN + ViT object tokenizer, and Part 4 on the 650K trajectory dataset from PyBullet. You now have a solid understanding of what VIMA is and what it achieves in a tabletop setting.

But one major question remains: can VIMA leave the table?

This is the series capstone — where we see how VIMA gets extended to high-DoF humanoid robots, the real challenges involved, and why this leap isn't as simple as changing a number in a config file.

Series Roadmap

This is Part 5 of 5 in the VIMA: Multimodal Prompts for Robot Manipulation series:

Vanilla VIMA — "Prisoner of the Table"

Recall the original VIMA setup: a 2-finger gripper robot arm, a flat PyBullet tabletop environment, and an action space of SE(2) — a 2D plane on the table surface.

Each VIMA action consists of two parts:

• Pick pose: (x, y, rotation) — where the gripper picks the object
• Place pose: (x, y, rotation) — where the gripper places the object

In practice this is 4 values: (x, y, rotation, open/close) for the gripper. The z height is hard-coded — the robot never decides "at what height to grasp" or "how to tilt the arm." The simulator handles that via inverse kinematics.

This is exactly why VIMA trains fast and achieves high performance: the problem is maximally simplified. But it's also why you can't deploy vanilla VIMA on a real robot without significant modification.

An analogy: tabletop manipulation is like learning to drive on a straight road in a 2D simulation — you only choose "turn left" or "turn right." A real humanoid robot is like riding a bicycle over 3D terrain with 23 muscles coordinating simultaneously.

What Does a Real Humanoid Actually Need?

Unitree G1 — 23 DoF

The Unitree G1 is one of the most widely used humanoids in research today, with 23 degrees of freedom distributed as:

• Each leg: 6 DoF × 2 = 12 DoF (hip pitch/roll/yaw, knee, ankle pitch/roll)
• Waist: 1 DoF (torso rotation)
• Each arm: 5 DoF × 2 = 10 DoF (shoulder pitch/roll/yaw, elbow, wrist)

The low-level controller runs at 200 Hz using a pure PD controller:

τ = Kp(q_target - q) - Kd × q̇

Where q is the current joint angle, q_target is the target angle, and τ is the motor torque. Our policy runs at 50 Hz, outputting q_target — meaning 23 joint position deltas every 20ms timestep.

Direct comparison:

AgiBot X2 — 25–31 DoF

AgiBot X2 (open-source humanoid from AgiBot) comes in multiple variants with different dexterity levels:

• X2 Lite: ~27 DoF, 5-DoF arms
• X2 Standard: 25 DoF
• X2 Pro: ~31 DoF, 7-DoF arms (more dexterous)
• X2 Ultra: 30 DoF, 3 kg payload, 558 mm arm reach

With 7-DoF arms like the X2 Pro, each arm can "thread through" narrow spaces — critical for precise assembly or manipulation in constrained environments. This is a key improvement over the traditional 6-DoF industrial robot arm.

What G1 and X2 share: an action space vastly larger than vanilla VIMA, and most real-world tasks require whole-body coordination — legs maintaining balance, torso leaning to extend reach, and both arms working together. Not just a single arm operating independently like in tabletop.

Strategy: Freeze Backbone, Swap Action Head

When adapting VIMA for humanoids, the first question is: what do we change, and what do we keep?

VIMA has three main components (analyzed in detail in Part 1):

1. T5 Encoder — encodes the multimodal prompt (text tokens + image object tokens)
2. XAttn GPT Decoder — cross-attention transformer processing observation history + prompt
3. Action Head — MLP decoding the output token into a concrete action

Clear reasons to freeze T5 + XAttn GPT:

• T5 has learned rich semantic representations from billions of text tokens — this is VIMA's core strength. Understanding "place the red cube at the target" doesn't depend on whether the robot has 6 or 23 DoF.
• Cross-attention has learned to link prompts with observations — it knows how to "look at" relevant object tokens when processing each word in the prompt. This capability transfers to new domains.
• Training T5 + XAttn GPT takes GPU-days; with limited humanoid data (thousands vs. 650K trajectories), you will immediately overfit if you fine-tune the full model.

The only part that needs to change is the action head — instead of decoding into a 4D end-effector pose, we need to decode into 23D joint position deltas.

Code: Custom 23-DoF Action Head

import torch
import torch.nn as nn

class VIMAActionHead6DoF(nn.Module):
    """Original VIMA action head: end-effector 6-DoF.
    
    Output: pick pose (3D position + 4D quaternion) and place pose.
    In tabletop: effectively only x, y, rotation (z is fixed).
    """
    def __init__(self, d_model: int = 512):
        super().__init__()
        # VIMA uses two separate MLPs: position (3D) and rotation (4D quaternion)
        self.pos_mlp = nn.Sequential(
            nn.Linear(d_model, 256),
            nn.ReLU(),
            nn.Linear(256, 3)   # xyz
        )
        self.rot_mlp = nn.Sequential(
            nn.Linear(d_model, 256),
            nn.ReLU(),
            nn.Linear(256, 4)   # quaternion (w, x, y, z)
        )

    def forward(self, x):
        # x: (batch, seq_len, d_model)
        return self.pos_mlp(x), self.rot_mlp(x)


class VIMAActionHead23DoF(nn.Module):
    """New action head: 23 joint position deltas for Unitree G1.
    
    Fully replaces the original action head — T5 + XAttn GPT stay frozen.
    Output: delta_q for 23 joints (12 legs + 1 waist + 10 arms)
    """
    def __init__(
        self,
        d_model: int = 512,
        n_joints: int = 23,
        action_scale: float = 0.05,  # small scale to avoid joint jumps at init
    ):
        super().__init__()
        self.n_joints = n_joints
        self.action_scale = action_scale

        # Single MLP: d_model → 512 → 256 → n_joints
        self.joint_mlp = nn.Sequential(
            nn.Linear(d_model, 512),
            nn.ReLU(),
            nn.Dropout(p=0.1),
            nn.Linear(512, 256),
            nn.ReLU(),
            nn.Linear(256, n_joints)
        )

        # Zero-init the final layer: robot stands still at training start
        nn.init.zeros_(self.joint_mlp[-1].weight)
        nn.init.zeros_(self.joint_mlp[-1].bias)

    def forward(self, x):
        # x: (batch, seq_len, d_model)
        joint_deltas = self.joint_mlp(x)         # (batch, seq_len, 23)
        return joint_deltas * self.action_scale   # scale down so PD controller isn't shocked

Key technical point: zero-initialize the final layer. At the start of fine-tuning, the policy outputs delta = 0 (robot stands perfectly still), gradually learning to move each joint. This is far safer than random initialization, which could cause erratic movement immediately — critically important with real hardware that can be damaged.

Training Config

import torch

# 1. Load pre-trained VIMA (T5 + XAttn GPT + old action head)
vima_model = load_pretrained_vima("VIMA-200M.ckpt")

# 2. Freeze the entire backbone — no gradient flows through T5 or XAttn GPT
for name, param in vima_model.named_parameters():
    if "action_head" not in name:
        param.requires_grad = False

# 3. Swap action head: 4D end-effector → 23D joint deltas
vima_model.action_head = VIMAActionHead23DoF(
    d_model=512,
    n_joints=23,
    action_scale=0.05
)

# 4. Optimizer only sees the new action head
optimizer = torch.optim.AdamW(
    filter(lambda p: p.requires_grad, vima_model.parameters()),
    lr=1e-4,
    weight_decay=1e-5
)
scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(
    optimizer, T_max=10_000, eta_min=1e-6
)

# 5. Loss: MSE between predicted deltas and expert deltas from teleoperation
criterion = torch.nn.MSELoss()

# Check trainable parameter count:
trainable = sum(p.numel() for p in vima_model.parameters() if p.requires_grad)
total = sum(p.numel() for p in vima_model.parameters())
print(f"Trainable: {trainable:,} / {total:,} ({100*trainable/total:.2f}%)")
# Example with VIMA-200M: Trainable: 141,335 / 200,141,335 (0.07%)

Only ~0.07% of parameters are trained. This is why the strategy works with limited humanoid data: fewer params = less data needed to converge without overfitting the backbone.

Data Gap — The Unavoidable Challenge

This is the largest problem when moving from tabletop to humanoid: the data gap.

VIMA was trained on 650K trajectories — all from PyBullet simulation, all tabletop tasks with a simple arm and oracle scripts. For humanoid manipulation, the picture is completely different:

The problem isn't just quantity. Humanoid manipulation data is harder to collect along three dimensions:

1. Expensive to collect: Requires teleoperation with complex dual-arm systems — exoskeleton suits, VR controllers, or puppet arms. Each hour of data can cost dozens of hours of setup and operation.

2. Low diversity: Each VIMA task has ~50K trajectories (oracle scripts generate infinite variations automatically). Humanoid datasets typically have a few thousand trajectories per task, with far fewer variations.

3. Noisy from human operators: VIMA's oracle scripts produce perfect demonstrations along optimal paths. Human teleoperators of real humanoids introduce jitter, hesitation, and sub-optimal paths — significantly noisier data.

An analogy: if VIMA-tabletop is like teaching a child to move their fingers on a 2D surface with 650,000 practice sessions, humanoid adaptation is like teaching that same child to play cello — the musical knowledge (prompt understanding) transfers, but 23 muscles and joints need to learn entirely new coordination patterns, with only a few thousand practice sessions available.

Approach 1: Teacher-Student Distillation

One of the most promising strategies to bridge the data gap is teacher-student distillation — leveraging VIMA's knowledge to train a humanoid-specific policy.

Core idea: Vanilla VIMA (teacher) has learned extremely well to understand multimodal prompts and plan manipulation tasks — even if only in tabletop. Knowledge about "this is a pick-and-place task" and "the red block needs to reach this target" is semantic task understanding that doesn't depend on robot morphology.

How it works in three steps:

1. Teacher (frozen VIMA): observes the same scene and outputs an action distribution π_teacher(a | s, prompt)
2. Student (smaller humanoid policy): also observes the scene but outputs π_student(q_delta | s, prompt) — joint deltas in 23D action space
3. Distillation loss: student learns to "think like VIMA" about task planning while "acting like a humanoid" for execution:

# Combined loss: imitation from expert + alignment with teacher reasoning
loss_imitation = criterion(predicted_deltas, expert_deltas)  # from teleoperation data
loss_distil = kl_divergence(student_latent, teacher_latent)  # align intermediate reps

loss_total = loss_imitation + alpha * loss_distil  # alpha = 0.1 typically works well

A closely related paper is Refined Policy Distillation (RPD, 2025): distill a VLA policy into an RL student, then let the student self-improve via reinforcement learning to surpass the teacher — since the student is no longer constrained by imitation loss and can explore better solutions.

Practical advantages: The student is much smaller (potentially a few million vs. 200M params), inference is fast enough to hit 50 Hz on embedded GPUs like the Jetson Orin in G1, and it can specialize to each robot's specific morphology and sensor configuration.

Approach 2: LoRA Fine-tuning

Instead of fully freezing XAttn GPT, another option is Low-Rank Adaptation (LoRA) — inserting small adapter matrices inside transformer layers without changing the original weights:

from peft import LoraConfig, get_peft_model, TaskType

# Apply LoRA only to XAttn GPT, keep T5 completely frozen
lora_config = LoraConfig(
    r=16,               # adapter matrix rank
    lora_alpha=32,      # scaling factor (usually 2*r)
    target_modules=[    # only attention projections, not FFN layers
        "q_proj",
        "v_proj",
        "cross_attn.q_proj",
        "cross_attn.v_proj",
    ],
    lora_dropout=0.05,
    bias="none",
    task_type=TaskType.FEATURE_EXTRACTION
)

# Apply LoRA to VIMA's XAttn GPT decoder
xattn_gpt_with_lora = get_peft_model(vima_model.xattn_gpt, lora_config)
xattn_gpt_with_lora.print_trainable_parameters()
# Example: trainable params: 1,572,864 || all params: 124,439,552 (1.26%)

With LoRA, the XAttn GPT can lightly adapt to the humanoid domain: relearning how to attend to visual tokens from multiple cameras (head + wrist), how to handle a more complex observation space — while preserving the majority of pretrained knowledge.

When to use which approach:

Performance Estimates

Based on synthesizing results from related papers on transfer learning for robot manipulation:

Note: these are estimates synthesized from the literature, not official ablation results on a VIMA-humanoid system. Real numbers depend heavily on the quality, diversity, and collection method of humanoid training data.

Sim2Real for Humanoids — Practical Domain Adaptation

Even after successful simulation fine-tuning, a significant gap remains between sim and the real robot — especially for humanoids because the dynamics are more complex, the number of contacts is higher, and contact-rich manipulation is hard to simulate accurately.

Three common techniques to bridge the sim2real gap:

1. Domain Randomization (DR): Randomize physical parameters during sim training — object mass and inertia, friction coefficients, lighting and color, camera noise, actuator delay. The policy is forced to learn robust behaviors rather than overfitting to specific simulator physics.

Example with Isaac Lab:

# Randomize joint PD gains so the policy is robust to actuator variation
joint_kp_randomizer = UniformNoise(
    operation="scale",
    distribution_parameters=[0.8, 1.2]  # ±20% Kp variation
)
scene.add_randomizer("joint_kp", joint_kp_randomizer)

2. Privileged Teacher Learning: Train a "teacher" policy in sim with full information — ground-truth state, perfect dynamics, no noise. The student policy only receives sensor observations like a real robot: RGB images, IMU, joint encoders. The student learns from the teacher via distillation while the teacher helps navigate difficult parts of the state space.

This approach was used successfully by SENTINEL (2024) and many humanoid locomotion papers — the final student policy is often more robust than the teacher because it learns to handle noise and uncertainty.

3. Adaptive Dynamics Estimation: After deploying on the real robot, use a small estimator to compute actual dynamics parameters (payload, joint friction, contact stiffness) and pass them to the policy as an additional context vector. The policy trains in sim with many "context versions," then at deployment time estimates the real context and uses it accordingly.

Multi-Camera Fusion and Latency Challenges

A technical challenge often overlooked is multi-camera fusion in humanoid settings.

Vanilla VIMA uses 2 fixed cameras (front + top) with a good field of view for tabletop tasks. A real humanoid has:

• Head camera: wide FOV, stable for locomotion (but changes when the robot bends forward)
• Wrist camera(s): close-up view, critical for fine manipulation (but shakes with arm motion)
• Depending on the robot: stereo cameras, depth sensors, or fisheye cameras

To integrate multiple cameras into VIMA's object tokenizer pipeline:

class MultiCamObjectTokenizer(nn.Module):
    """Extends VIMA's object tokenizer for multi-camera humanoid."""
    def __init__(self, n_cameras: int = 2, d_token: int = 256):
        super().__init__()
        self.n_cameras = n_cameras
        # Shared weights for detector and encoder across cameras
        self.obj_detector = MaskRCNN(pretrained=True)   # shared
        self.vit_encoder = ViTObjectEncoder(d_token=d_token)  # shared
        # Camera embedding to distinguish tokens from different cameras
        self.cam_embed = nn.Embedding(n_cameras, d_token)

    def forward(self, images: list, cam_ids: list):
        all_tokens = []
        for img, cam_id in zip(images, cam_ids):
            objects = self.obj_detector(img)
            tokens = self.vit_encoder(objects)   # (n_obj, d_token)
            cam_emb = self.cam_embed(torch.tensor(cam_id))
            tokens = tokens + cam_emb            # add camera identity embedding
            all_tokens.append(tokens)
        return torch.cat(all_tokens, dim=0)      # concat all object tokens

On latency: the 50 Hz requirement (20ms/step) is tight for a 200M-param transformer. Practical optimizations:

• Cache the T5 encoder output (prompt doesn't change within an episode → run T5 once per task)
• Use TensorRT or torch.compile to optimize XAttn GPT inference
• Consider distilling to a smaller student (tens of M params) if deploying on Jetson Orin NX

Series Conclusion — VIMA as a Building Block

After 5 parts, we've gone from "what is VIMA" to "how to bring VIMA to humanoid robots." Looking back at what we learned:

Part 1 — Architecture: Cross-attention isn't a random design choice — it matters because prompt conditioning happens at every decoder layer, not just at the prefix like GPT-style. This is the foundation of zero-shot generalization capability. And this knowledge transfers fully to the humanoid domain.

Part 2 — Benchmark: 17 tasks and 4 generalization levels (L1→L4) is a more rigorous evaluation framework than most robot learning benchmarks. When adapting to humanoids, L4 (novel task) performance is the most honest indicator.

Part 3 — Object Tokenizer: Mask R-CNN + ViT transforms raw images into object tokens — this is why 1% of data beats all baselines. Object-centric representation learns semantics better than pixel-level approaches. When adapting to multi-camera humanoid, we only need to extend the tokenizer, not replace it.

Part 4 — Dataset: 650K trajectories from PyBullet with oracle scripts are a strong foundation because of broad coverage. But this is also the biggest weakness when adapting to humanoids — there's no equivalent 650K humanoid trajectory dataset to train from scratch.

Part 5 (this part) — Humanoid Adaptation: Freeze backbone (T5 + XAttn GPT), swap action head from 4D → 23D joint deltas. Data gap is challenge #1. Teacher-student and LoRA are the two most viable approaches today.

The Road Ahead

VIMA is a strong building block for multimodal prompt robots — but to truly escape the tabletop and work effectively on humanoids, the ecosystem needs three things still under active development:

1. Large-scale humanoid data. Datasets like Hy-Embodied UMI with 2,000 hours of bimanual data and AgiBot World are building this foundation — but need to scale another 10–100× to match the 650K tabletop trajectories in diversity and coverage.

2. Appropriate action representation for whole-body. Joint delta is a good starting point, but tasks requiring leg-arm coordination (picking something off the floor while walking) demand a more complex action space — possibly hierarchical: high-level goal (end-effector target) + low-level joint control, with a whole-body controller layer in between.

3. Better sim2real pipelines for contact-rich manipulation. Isaac Sim and MuJoCo are narrowing the physics gap, but manipulation with many contact points (screwing, assembly, weaving fabric) remains a major challenge that simulation cannot yet accurately model.

Those following this field are witnessing a clear trend: multimodal prompting (VIMA's core idea) is becoming the standard interface for robot manipulation. The difference between modern papers is no longer "whether to use multimodal prompts" but "how many DoF, how much data, how well the sim2real gap is bridged, and whether inference is fast enough for real-time deployment."

From tabletop 4D to humanoid 23D is not just adding a number — it's a leap that requires rethinking the entire data pipeline, action representation, and deployment strategy. VIMA gives us a strong starting point: a pretrained backbone with excellent multimodal understanding. The rest — data, action space, sim2real — is the work ahead for the global robotics research community.

Summary

• Vanilla VIMA: tabletop, SE(2) action space 4D, single 2-finger gripper arm
• Unitree G1: 23 DoF (12 legs + 1 waist + 10 arms), 200 Hz PD controller, 50 Hz policy
• AgiBot X2: 25–31 DoF depending on variant; X2 Pro has 7-DoF dexterous arms
• Adaptation strategy: freeze T5 + XAttn GPT, swap action head 4D → 23D joint deltas
• Safe initialization: zero-init final action head layer to avoid joint shock
• Data gap: humanoid data = thousands of traj vs. 650K tabletop — challenge #1
• Frozen approach: ~0.07% trainable params, ~4 hours on 5K trajectories
• LoRA approach: ~0.8% + head trainable, lightly adapts backbone, better balance
• Teacher-student: VIMA as teacher → smaller, faster student suitable for real hardware
• Sim2real: domain randomization + privileged teacher + adaptive dynamics
• Multi-camera: camera embedding to distinguish head cam vs. wrist cam in tokenizer
• Latency: cache T5 prompt encoding, use TensorRT/compile to hit 50 Hz target
• Series conclusion: VIMA is a strong building block; needs large-scale humanoid data to truly generalize beyond the tabletop

Related Posts

• Part 4: Dataset 650K — Large-Scale Multi-Task Data Collection for VIMA
• Part 1: Cross-Attention Architecture — The Heart of VIMA
• Hy-Embodied-0.5-VLA Guide: Bimanual VLA with 2,000 Hours of UMI Data