MiniVLA V3 on MuJoCo: Robust VLA

MiniVLA is a practical branch of the OpenVLA family: it keeps the vision-language-action idea, reduces the model footprint from a 7B-scale VLA to roughly 1B parameters, uses Qwen 2.5 0.5B as the language backbone, adds Vector Quantized action chunking, and supports richer visual inputs such as image history or wrist camera frames. That combination makes MiniVLA interesting for manipulation engineers because smaller models are easier to iterate, benchmark, debug, and deploy.

This guide walks through the full beginner-friendly workflow: the paper idea, architecture, installation, LIBERO/MuJoCo evaluation, robustness perturbations, training, inference, and results. The requested topic includes "MiniVLA V3" and a 97.92% success rate. A precision note matters here. In the public sources I could verify, the official Stanford work is named MiniVLA, MiniVLA VQ h8, MiniVLA history=2, or MiniVLA wrist images. I did not find an official paper named "MiniVLA V3" that directly reports 97.92% on MuJoCo. In this article, "V3" is therefore used as a practical configuration name: MiniVLA 1B + VQ action chunking + multi-image/history + robustness evaluation on LIBERO/robosuite/MuJoCo. The 97.92% number is treated as a target or reported-run metric that must be verified from your own rollout logs, for example 47/48 successful trials or 235/240 successful trials, not as an unverified claim assigned to the original paper.

Primary sources:

• Stanford SAIL Blog: MiniVLA: A Better VLA with a Smaller Footprint
• Original GitHub repo: Stanford-ILIAD/openvla-mini
• Edge/TensorRT implementation: Zhenxintao/MiniVLA
• Weights: xintaozhen/MiniVLA and Stanford-ILIAD/minivla-vq-libero90-prismatic
• Benchmark: LIBERO, running on robosuite/MuJoCo
• Simulator layer: robosuite and MuJoCo

If you are new to VLA policies, start with VLA and LeRobot: the framework view. If MuJoCo itself is new to you, read Getting Started with MuJoCo before debugging OpenGL, camera, XML asset, or offscreen rendering errors.

The Paper Idea: Smaller Does Not Mean Less Structured

OpenVLA showed that a VLM-style model can generate robot actions from images and language. The problem is cost. A 7B model is heavy for training, heavy for inference, and slow to iterate in academic or small-company settings. MiniVLA asks a pragmatic question: if we keep the OpenVLA/Prismatic visual stack but replace the large language backbone with Qwen 2.5 0.5B, can a 1B-scale VLA remain competitive?

The Stanford SAIL blog reports that MiniVLA reduces OpenVLA's footprint from 7B to about 1B parameters. On LIBERO-90, the basic MiniVLA model is close to the OpenVLA baseline: OpenVLA reports 62% success and MiniVLA reports 61.4% when trained to 95% action accuracy. That result is not the whole story, but it establishes the central point: the smaller model is not automatically unusable.

The more important contribution is the action representation. A basic OpenVLA-style action tokenizer discretizes each action dimension into bins. That is simple, but manipulation rarely works as isolated one-step decisions. Closing a drawer, picking a bowl, or placing a mug requires a short coherent trajectory: approach, contact, grasp or push, move, release, and recover. MiniVLA uses Residual VQ to compress a short action chunk (H x A) into codeword indices. The VLA then predicts tokens representing learned action chunks instead of only independent scalar bins. In the Stanford results, MiniVLA + VQ h8 improves LIBERO-90 success from 61.4% to 77%.

The third step is multi-image input. A tabletop robot often needs hidden temporal information: whether the gripper already touched the object, whether the drawer moved, whether the object slipped, or whether a previous action changed the scene. A single image can miss that. MiniVLA adds history=2 or wrist images, and the blog reports 82% for MiniVLA VQ h8 + history=2 and 82.1% for MiniVLA VQ h8 + wrist images on LIBERO-90. That is why this guide uses "V3" as a practical shorthand for the stronger setup: small backbone, chunked actions, richer visual context, and a proper robustness loop.

Why LIBERO Counts as MuJoCo

Beginners often see run_libero_eval.py and assume the benchmark is separate from MuJoCo because it does not directly open a raw mujoco.viewer script. The actual stack is layered. LIBERO defines language-conditioned manipulation tasks. robosuite provides the robot simulation framework, task assets, controllers, cameras, and rendering. robosuite is powered by the MuJoCo physics engine. So when MiniVLA is evaluated on LIBERO, the policy is being tested in a MuJoCo-based manipulation environment.

That matters for robustness. A policy can look strong under the default seed and still fail when the evaluation distribution shifts slightly:

• object pose shifts by a few centimeters;
• camera crop changes;
• lighting or renderer settings shift;
• action horizon does not match the VQ tokenizer;
• gripper threshold or controller frequency is wrong;
• the language command is paraphrased;
• a drawer or cabinet contact requires force rather than only a visually plausible motion.

Do not treat a single success rate as universal manipulation ability. Treat it as the result of a specific evaluation protocol: task suite, number of trials, random seeds, camera setup, checkpoint, perturbations, and success predicate.

Runtime Architecture

The execution pipeline has four practical layers:

Instruction + RGB image/history/wrist
        |
        v
Vision encoder: DINOv2 / SigLIP style ViT
        |
        v
Qwen 2.5 0.5B language backbone inside Prismatic VLM
        |
        v
Action tokenizer:
  - base binning for single action, or
  - VQ h8 chunk tokenization for short horizon
        |
        v
LIBERO environment on robosuite/MuJoCo
        |
        v
success/failure, rollout video, latency, perturbation logs

The Zhenxintao/MiniVLA repo adds a deployment-oriented layer: exporting the vision encoder to ONNX/TensorRT and serving it through a microservice. Its README reports a PyTorch baseline around 4115 MiB on an RTX 4060 Laptop GPU with 8 GB VRAM, while vision-only TensorRT lowers memory to around 3892 MiB and end-to-end latency from 365 ms to 274 ms. Full Vision+LLM TensorRT lowers latency further but yields invalid outputs and 0% success in the reported ablation. The engineering lesson is straightforward: optimize only after the action path is correct. For robustness, a slightly slower correct policy is better than a faster invalid one.

Machine Setup

Use Ubuntu 22.04, Python 3.10, CUDA 12.x, and an NVIDIA GPU. For lightweight inference and evaluation, an 8 GB GPU can be workable. For full training or training from scratch, expect much more compute. A beginner should not start with full training. A safer path is:

1. install the environment;
2. download existing checkpoints;
3. run 5-10 sanity trials;
4. run a 48-trial or 240-trial robustness check;
5. fine-tune only after the evaluation stack is clean.

Basic installation:

conda create -n minivla python=3.10 -y
conda activate minivla

conda install pytorch torchvision torchaudio pytorch-cuda=12.4 -c pytorch -c nvidia -y

git clone https://github.com/Stanford-ILIAD/openvla-mini.git
cd openvla-mini
pip install -e .

pip install packaging ninja
pip install "flash-attn==2.5.5" --no-build-isolation

git clone https://github.com/Lifelong-Robot-Learning/LIBERO.git
cd LIBERO
pip install -e .

cd ../openvla-mini
pip install -r experiments/robot/libero/libero_requirements.txt

If you want to test the Zhenxintao/MiniVLA TensorRT service later:

git clone https://github.com/Zhenxintao/MiniVLA.git
cd MiniVLA
pip install -e .
cd tensorRT-scripts
python tensorRT_vision_service.py

Start with the PyTorch baseline. Do not enable TensorRT first. You need to prove that the checkpoint, VQ path, LIBERO data, camera observations, and rendering are correct before changing the perception backend.

Download Checkpoints and VQ Tokenizer

A MiniVLA VQ setup needs two pieces: the Prismatic checkpoint and the VQ-VAE action tokenizer.

mkdir -p ~/models
cd ~/models
git lfs install
git clone https://huggingface.co/Stanford-ILIAD/minivla-vq-libero90-prismatic

cd ~/openvla-mini
mkdir -p vq
huggingface-cli download Stanford-ILIAD/pretrain_vq \
  --local-dir ./vq \
  --local-dir-use-symlinks False

Expected structure:

~/models/minivla-vq-libero90-prismatic/checkpoints/*.pt
~/openvla-mini/vq/pretrain_vq+mx-libero_90+fach-7+ng-7+nemb-128+nlatent-512/
  checkpoints/model.pt
  config.json

If the VQ tokenizer is missing or mismatched, the failure may not look like a clean exception. The model may output tokens, but those tokens decode to incorrect chunks. In manipulation, that can mean wrong action scale, inverted gripper behavior, late contact, or a robot that appears alive but never completes the task.

Smoke Test LIBERO/MuJoCo

Run a very small evaluation first:

python experiments/robot/libero/run_libero_eval.py \
  --model_family prismatic \
  --pretrained_checkpoint ~/models/minivla-vq-libero90-prismatic/checkpoints/step-150000-epoch-67-loss=0.0934.pt \
  --task_suite_name libero_90 \
  --center_crop True \
  --hf_token $HF_TOKEN \
  --num_trials_per_task 1

The goal of this smoke test is not a high success rate. The goal is to verify that:

• MuJoCo/robosuite can render offscreen images;
• the checkpoint loads;
• the VQ tokenizer path is correct;
• the policy emits the expected 7-DoF action format;
• rollout videos are saved;
• success and failure are logged clearly.

If you see GL, EGL, mujoco_py, robosuite==1.4.0, blank camera, or missing asset errors, stop. Do not train. Do not rewrite prompts. Fix the simulator first. A VLA policy evaluated on broken images is not being evaluated at all.

Designing a Robustness Test for 97.92%

The number 97.92% is easy to misuse. It corresponds to several possible trial counts:

For robotics, 47/48 is a useful quick check but not a strong general robustness claim. It is evidence on a narrow task set. For an internal technical report, use at least 240 trials split across tasks, seeds, perturbations, and language paraphrases. A beginner-friendly protocol can look like this:

Tasks:
  8 LIBERO desktop manipulation tasks

Trials:
  6 seeds per task = 48 trials for quick check
  30 seeds per task = 240 trials for stronger internal report

Perturbations:
  P0: default scene
  P1: object xy shift ±2 cm
  P2: object yaw shift ±10 deg
  P3: camera crop jitter 0.90-1.00
  P4: language paraphrase
  P5: distractor object present

Metrics:
  success_rate
  mean_steps_to_success
  failure_mode
  invalid_action_rate
  gripper_misfire_rate
  wall_clock_latency_ms

Do not log only success=True or success=False. Log the failure mode. A policy that never touches the object failed for a different reason than a policy that grasps correctly but cannot push the drawer far enough. MiniVLA-style failures often involve action chunks overshooting, camera crop mismatch, late gripper closure, or the policy getting stuck in a repeated sub-step because image history is not ordered the same way as training.

Adding Lightweight Perturbations

Depending on your LIBERO version, you may implement perturbations through a wrapper around environment reset, task initial state, or camera preprocessing. Start with changes that do not require deep XML edits:

import random
import numpy as np

def sample_eval_case(seed: int):
    rng = random.Random(seed)
    return {
        "seed": seed,
        "object_xy_jitter": np.array([
            rng.uniform(-0.02, 0.02),
            rng.uniform(-0.02, 0.02),
        ]),
        "object_yaw_jitter_deg": rng.uniform(-10.0, 10.0),
        "center_crop": rng.choice([True, True, False]),
        "language_variant": rng.choice([
            "put the black bowl in the tray",
            "pick up the black bowl on the left and place it in the tray",
            "move the left black bowl into the tray",
        ]),
    }

Then pass the seed into environment reset, replace the instruction if the evaluation script allows it, and save a JSONL log:

{"task":"black_bowl_to_tray","seed":17,"perturb":"P1+P4","success":true,"steps":142,"latency_ms":281}
{"task":"black_bowl_to_tray","seed":18,"perturb":"P1+P4","success":false,"steps":400,"failure":"gripper_closed_late"}

If you have never wrapped LIBERO before, read experiments/robot/libero/run_libero_eval.py first. Find where the benchmark suite is created, where the task is selected, where the environment resets, where observations are converted to policy inputs, where actions are stepped, and where success is checked. Intervene around reset and language instruction first. Do not edit the model before you have a baseline.

Training and Fine-Tuning

There are three training levels:

The OpenVLA-Mini README gives this example for MiniVLA training on LIBERO-90:

torchrun --standalone --nnodes 1 --nproc-per-node 8 vla-scripts/train.py \
  --vla.type "prism-qwen25-dinosiglip-224px+0_5b+mx-libero-90" \
  --data_root_dir <PATH_TO_LIBERO_DATA_ROOT> \
  --run_root_dir <PATH_TO_LOG_CHECKPOINT_ROOT> \
  --wandb_project "<PROJECT>" \
  --wandb_entity "<ENTITY>"

For VQ action chunking, the VQ model must be trained or downloaded first. The VQ model learns a codebook for action chunks; the VLA then learns to predict codeword tokens. If you change robot embodiment, action dimension, control frequency, or action horizon, do not blindly reuse the old VQ tokenizer. LIBERO's 7-DoF delta-pose-plus-gripper action space is not the same as a joint-position robot, a bimanual arm, or a mobile manipulator.

If your goal is 97.92% on a narrow task set, fine-tuning is usually more useful than prompt engineering. But the data must cover the perturbations you plan to test. The OpenVLA-Mini troubleshooting notes also emphasize continuous demonstrations, limited idle actions, enough diversity in initial conditions, and a control frequency consistent with the model. With chunking, consistency between training frequency and inference frequency still matters.

Inference Loop

Conceptually, inference is simple:

obs = env.reset()
instruction = task.language

for t in range(max_steps):
    image = obs["agentview_image"]
    wrist = obs.get("robot0_eye_in_hand_image")
    state = obs.get("robot_state")

    action = policy.predict_action(
        image=image,
        instruction=instruction,
        state=state,
        do_sample=False,
    )

    obs, reward, done, info = env.step(action)
    if info.get("success", False):
        break

The real difficulty is preprocessing. You need the correct crop, image order, tokenizer, unnormalization key, and dtype. For multi-image MiniVLA, the README warns that images must be passed in the exact same order and count as training, because the visual token sequence is just concatenated as image1_tokens image2_tokens ...; there is no explicit camera tag. If a history=2 model receives only the current image at inference time, success can collapse without producing a clean runtime error.

Reading Results

Published Stanford MiniVLA results:

Zhenxintao/MiniVLA deployment ablation:

If your own log contains 47/48 successes, report it like this:

MiniVLA V3-style configuration:
  checkpoint: minivla-vq-libero90-prismatic
  eval: 8 tasks x 6 seeds
  simulator: LIBERO on robosuite/MuJoCo
  perturbations: object xy ±2 cm, yaw ±10 deg, paraphrase
  success: 47/48 = 97.92%

For a stronger claim, run 240 trials:

success_rate = 235 / 240 = 97.92%
95% Wilson confidence interval should be reported.

Even when the success rate is high, inspect the failure videos. One failure may reveal a serious issue: black objects under a lighting shift, late gripper closure, wrong drawer contact, or a policy that depends on a memorized camera crop. That information is more useful than the percentage alone.

Debug Checklist

For deeper policy stress testing, read StressDREAM for robot manipulation policy tests. For fine-tuning and deployment, see PEFT and VLA deployment.

Conclusion

MiniVLA is worth studying because it makes VLA manipulation more accessible: smaller than OpenVLA, faster to iterate, compatible with action chunking, able to use richer visual context, and supported by repos that can run real evaluations. LIBERO on robosuite/MuJoCo is a good place to test before risking hardware. But a number like 97.92% only matters when the task set, trial count, seeds, perturbations, checkpoint, success predicate, and failure logs are explicit.

For beginners, the right goal is not to chase a polished percentage first. The right goal is a repeatable evaluation pipeline: baseline runs, rollout videos look correct, perturbation logs are clear, action decoding is verified, and only then fine-tuning begins. Once that pipeline is clean, a high success rate becomes meaningful.

Related Posts

• VLA and LeRobot: the framework view
• Getting Started with MuJoCo
• StressDREAM for robot manipulation policy tests