Train GVLA with MiGA for Any Gripper

Why Gripper-Aware VLA Matters

If you have trained a pick-and-place policy with LeRobot, OpenVLA, or a pi0-style model, you have probably seen a quiet assumption: the robot sees an image, reads a language command, and predicts actions in a normalized action space. That assumption is usable when the dataset comes from one arm and one parallel-jaw gripper. It becomes much weaker when the end-effector changes. A suction cup often approaches a flat surface from above. A parallel-jaw gripper may need to slide a thin object toward the table edge before side-grasping it. A soft gripper can use compliance. A dexterous hand may need finger pre-shaping before contact.

The paper GVLA: Gripper-aware Vision Language Action Models targets exactly this missing piece. The authors ask whether VLA policies can learn embodiment dependence and strategy-level divergence across gripper morphologies. They contribute two things. First, MiGA, a multi-gripper-aware dataset with 103K demonstrations, 36 tasks, 5 gripper types, and both simulation and real-world data. Second, GVLA, a framework that adds gripper conditioning to a VLA backbone through a multi-gripper tokenizer and a dual Mixture-of-Adapters module.

This guide explains the idea, architecture, installation plan, training loop, inference loop, and reported results. The official project page currently provides the paper, figures, video, and a scripts link, while the full public dataset/checkpoint release may still be staged. So the implementation section is written as a practical lab recipe: use the official code and dataset when they are available, and use the same schema and adapter pattern with your own data while waiting.

Primary sources:

• arXiv paper: Gripper-aware Vision Language Action Models
• Project page: airvlab.github.io/G-VLA
• Project PDF: GVLA paper PDF
• Scripts link from the project page: airvlab/GCA-Bench

The Paper Idea in One Sentence

GVLA does not only ask "where is the object?" It also asks "given this gripper, what is the right way to approach and manipulate the object?" That is the key difference between a gripper-agnostic policy and a gripper-aware policy.

A conventional VLA model usually consumes image tokens, language tokens, and sometimes proprioception tokens. GVLA adds explicit conditioning for the end-effector. Instead of describing the gripper with a brittle text prompt such as "this is a suction gripper", the paper uses learnable soft prompts at three levels:

These tokens are concatenated and prepended to the observation embeddings. The transformer therefore receives gripper identity before it predicts actions. A model could try to infer the gripper from wrist images alone, but that signal is unreliable under cropping, occlusion, camera changes, and domain shift. GVLA gives the model a structured latent representation: platform tokens share robot-level knowledge, gripper-type tokens share contact affordances, and instance tokens keep smaller details such as jaw width, TCP offset, compliance, or suction cup diameter.

What Is Inside MiGA?

MiGA is not just a large mixed robot dataset. It is designed to reveal how different grippers solve the same task with different strategies. According to the paper, MiGA contains 103,000 demonstrations, 36 tasks, 5 gripper types, multi-view RGB-D observations, proprioceptive states, gripper-strategy annotations, natural language descriptions for every gripper-task pair, and about 5% failure demonstrations for studying embodiment limits.

The five gripper families are:

The task suite has four categories: singulated, stacked, constrained, and semantic. For a beginner, think of them as four tests. Singulated scenes test object geometry, texture, and pose. Stacked scenes test occlusion and collision constraints. Constrained-space tasks test whether gripper size, reachability, and alignment precision matter. Semantic tasks test whether the policy can reason about task context, such as choosing a safe contact region when handling a filled container.

Installation Plan

To reproduce GVLA in a lab, split the system into four layers: data, backbone, gripper conditioning, and deployment. Do not start by writing model code. Start with the dataset schema, because GVLA depends on clean gripper metadata.

A minimal Python environment looks like this:

conda create -n gvla python=3.10 -y
conda activate gvla
pip install torch torchvision torchaudio
pip install transformers accelerate datasets safetensors einops
pip install opencv-python pillow numpy scipy pandas
pip install h5py zarr tqdm wandb

If you want simulation similar to the paper, install NVIDIA Isaac Lab using the official Isaac Lab instructions. If your lab already fine-tunes a pi0/pi0.5 backbone or an OpenVLA-style backbone, keep that stack. GVLA is an architectural pattern. You do not need to replace every part of your training system. The critical requirement is that each sample returns vision, language, action, proprioception, and gripper identity:

sample = {
  "images": {
    "wrist": Tensor[C, H, W],
    "third_view": Tensor[C, H, W]
  },
  "language": "pick up the flat box",
  "proprio": Tensor[D],
  "actions": Tensor[H, A],
  "robot_platform": "ur10",
  "gripper_type": "suction",
  "gripper_instance": "ur10_suction_cup",
  "task_category": "singulated"
}

When the official MiGA release is available, preserve the authors' naming as much as possible. If you use an internal dataset, map your metadata to the same three-level structure: platform, type, and instance. Do not store only gripper_id. Without gripper_type, the model cannot know that two different instances may share the same affordance. Without platform, the adapter may confuse robot kinematic limits with gripper morphology.

Converting Data into a Trainable Format

Beginners often convert actions before defining frames. That is risky for any robot learning project and even more dangerous for multi-gripper learning. Define these fields first:

A typical conversion pipeline is:

raw logs
  -> synchronize timestamps
  -> crop/resize images
  -> normalize proprioception
  -> express actions in one chosen frame
  -> attach gripper metadata
  -> split train/val/test by task and object

For GVLA, splitting matters a lot. If you randomly split by frame, validation becomes too easy because frames from the same episode can leak into both train and validation. Split by episode. Then create additional held-out splits by object and by gripper. The paper evaluates zero-shot object generalization and few-shot adaptation, so you need at least one unseen object or unseen task group to measure transfer honestly.

GVLA Architecture in Practice

GVLA adds two modules to the VLA backbone. The first is the multi-gripper tokenizer. It creates soft prompt embeddings:

P(h) = [P(platform); P(gripper_type); P(instance)]
X_conditioned = [P(h); X_observation]

P(platform) learns robot-level information such as reachability, joint layout, and kinematic bias. P(gripper_type) learns contact affordances for a mechanism class. P(instance) learns smaller differences such as jaw width, compliance, TCP offset, and suction diameter. These prompts are not hard one-hot labels. They are vectors optimized with the policy.

The second module is dual Mixture-of-Adapters. The paper uses two routers: a platform-aware router and a gripper-aware router. Each router selects top-k adapter experts from its pool. Each adapter is a small bottleneck network: down-projection, GeLU, up-projection, then residual addition to the hidden activation. The intuition is simple: the backbone keeps shared vision-language-action knowledge, while adapters specialize the action pathway for robot and gripper variants.

Minimal pseudo-code:

obs_tokens = encode_vision_language(images, language, proprio)
prompt = concat(platform_prompt[r], type_prompt[g], instance_prompt[u])
x = concat(prompt, obs_tokens)

hidden = backbone.transformer(x)
action_hidden = hidden[:, action_positions]

platform_weights, platform_experts = platform_router(mean(platform_prompt[r]))
gripper_weights, gripper_experts = gripper_router(mean(concat(type_prompt[g], instance_prompt[u])))

action_hidden = action_hidden + apply_adapters(action_hidden, platform_experts, platform_weights)
action_hidden = action_hidden + apply_adapters(action_hidden, gripper_experts, gripper_weights)
actions = action_head(action_hidden)

The paper reports a layer probing analysis showing that gripper type sensitivity rises strongly near the final layer. Therefore, the authors insert MoA into the final action layer, where it can directly modulate action generation. This is a useful design lesson: prompt tokens influence representation early, while adapters specialize the final action path.

Training GVLA Step by Step

GVLA optimizes three losses:

For a small lab version, start with MSE imitation loss or diffusion/flow matching depending on your backbone. The two losses you should not drop are gripper prediction and load balance. Without them, prompts and adapters can become decorative metadata rather than useful computation.

A practical training schedule:

Stage 0: sanity check
  Train 1-2 tasks, 2 gripper types, and about 1K demos.
  Goal: overfit the training set and confirm actions have the right sign and frame.

Stage 1: shared backbone warmup
  Freeze most of the VLA backbone.
  Train prompts, adapters, and the action head.
  Use a higher learning rate for prompts/adapters than for any unfrozen backbone layer.

Stage 2: gripper-aware fine-tuning
  Unfreeze the last few layers if GPU memory allows it.
  Enable action loss, gripper loss, and load balance loss.
  Log prediction error, router usage, gripper classification accuracy, and rollout success.

Stage 3: adaptation
  Keep the backbone mostly frozen.
  Add few-shot data for a new task or a new gripper.
  Fine-tune the instance token and related adapters first, then expand if needed.

Common training failures:

Inference on a Real Robot

GVLA inference needs three metadata fields in addition to images and language: platform_id, gripper_type, and gripper_instance. If you swap the gripper but forget to update metadata, the policy may route through the wrong adapters. A deployment loop should run in a receding-horizon style:

1. Capture wrist and third-view images.
2. Read proprioception and gripper state.
3. Build the language instruction.
4. Attach platform/type/instance ids.
5. Run GVLA to predict an H-step action chunk.
6. Send the first K actions to the low-level controller.
7. Observe again and repeat until success or timeout.

You should place a safety layer between GVLA and the robot controller. GVLA predicts strategy-level action chunks, but the paper also reports failure cases involving kinematic infeasibility and physical gripper limits. Deployment should clamp velocity, enforce workspace bounds, validate IK, limit force or gripper commands, and stop when visual tracking fails. For suction, add vacuum pressure checks. For dexterous hands, check joint limits on every finger.

What Results Did the Paper Report?

In simulation, GVLA with the pi0.5 backbone achieves 66.00% average success rate, compared with 58.38% for the vanilla pi0.5 baseline. That is a 7.62 percentage-point improvement. With the pi0 backbone, GVLA reaches 48.13%, compared with 38.13% for vanilla pi0. Traditional two-stage grasping baselines such as AnyGrasp and GraspMAS collapse on flat and stacked scenes because they are closer to open-loop grasp pose systems than trajectory-level manipulation policies.

The tokenizer comparison is also important. GVLA reports PE 0.032, CAPD 1.34, and GCS 0.249, outperforming MLP, VQ-VAE, and simple learned-prompt representations. Linear probing rises from about 59% at layer L0 to 80% at layer L12, suggesting that gripper tokens are propagated through the network and remain recoverable in hidden states.

For real-world validation, the authors use a UR5 arm with a Robotiq 2F-85 gripper. They adapt to unseen tasks with only 10 demonstrations and 20K fine-tuning steps. GVLA outperforms the pi0.5 baseline across the real-robot tasks, supporting the claim that gripper-aware conditioning improves transfer across unseen platform-gripper configurations.

How to Apply GVLA in a Small Lab

If you do not have 103K demonstrations, do not try to copy the entire paper immediately. Build a smaller version with the same structure. Choose two clearly different grippers, for example a parallel-jaw gripper and a suction cup. Pick 4-6 tasks where strategy divergence is visible: picking a thin box, picking a cup, retrieving an object from a narrow tray, grasping a soft object, handling stacked objects, and selecting a semantic contact region. Collect 50-100 demonstrations per task per gripper. That is enough to test the tokenizer, router behavior, and inference loop.

As you scale, prioritize controlled diversity over noisy volume. Every task should have a stable instruction template, camera calibration version, robot platform id, gripper type id, and action frame. If multiple operators collect data, log operator id so you can analyze bias later. If you capture failure demonstrations, do not throw all of them away. Label them and use them for analysis or for a future verifier.

GVLA is useful because it makes a practical point: a generalist robot policy must generalize not only across objects and language, but also across hardware. For robotics engineers, the engineering lesson is that dataset schema and end-effector metadata matter as much as model size. A large VLA that cannot tell whether it is controlling a suction cup, a three-finger gripper, or a dexterous hand may learn an average strategy. On real robots, average strategies often fail.

Related Posts

• YUBI gripper for LeRobot VLA
• OpenVLA deep dive
• FineVLA dual-arm VLA tutorial