ManiGuard-Bench for Safe VLA Eval

Why ManiGuard-Bench Matters

If you have fine-tuned a Vision-Language-Action (VLA) policy with LeRobot, openpi, GR00T, or SmolVLA, the first metric you usually check is task success. Did the robot pick the object? Did it place the jar in the target area? Did it close the lid? For real robot manipulation, that question is necessary but incomplete. A rollout can complete the goal while tipping a nearby glass, dropping the object once, lifting a container before it is closed, or touching a region that should stay clean.

ManiGuard addresses exactly this gap. The original paper, "MANIGUARD: A Benchmark and Data Suite for Specification-Grounded Safety Evaluation and Improvement of Robotic Manipulation", was posted to arXiv on August 18, 2026 by Yiyan Peng, Philip Wang, Simon Sinong Zhan, and collaborators. The official code is at NU-IDEAS-Lab/ManiGuard, the documentation is at nu-ideas-lab.github.io/ManiGuard, and the benchmark datasets plus evaluated checkpoints are hosted on Hugging Face.

The central idea is simple: task success and safety are independent axes. A policy should not receive full credit just because the final goal predicate is true. It should be credited only when it completes the task without violating a safety specification during the trajectory. ManiGuard-Bench therefore asks not only "did the robot finish?" but also "did the robot remain safe while finishing?".

This guide connects well with previous vnrobo articles such as SmolVLA on LeRobot, GR00T N1.6 fine-tuning, and SafeContract for VLA monitoring. Those articles focus on training and runtime guardrails. ManiGuard adds a reproducible evaluation layer with formal safety specifications and the same task definitions across openpi, GR00T, and SmolVLA.

What the Paper Proposes

ManiGuard has two tightly connected parts.

The first part is ManiGuard-Bench, a frozen benchmark for contact-rich household manipulation. It contains 200 base tasks across six task families: clutter_pickup, cabinet_pickup, stack_retrieve, jar_transport, lid_transport, and dusty_transfer. Each base task has one in-distribution version and four single-axis out-of-distribution perturbations: target appearance, language, location, and environment. This gives 1,000 locked evaluation scenarios. "Locked" matters here: the scene, goal, target object, prompt, and safety specification are fixed so model comparisons are not moving targets.

The second part is a safety-annotated data suite. The authors release 8,000 demonstrations, or 40 demonstrations for every base task. The demonstrations come from an automated motion-planning pipeline using cuRobo and from human teleoperation. The useful part is not only the action data. Every step is annotated by the same runtime safety monitor used during evaluation, so the data can support safety-aware supervised fine-tuning rather than plain imitation.

The major technical difference from a normal manipulation benchmark is that safety is written as an LTLf specification. LTLf means Linear Temporal Logic over finite traces. For beginners, think of it as a precise way to write rules such as "always keep the object upright", "do not lift the object before the lid is closed", or "after touching a dirty object, do not touch the clean target area". ManiGuard compiles this formula into a deterministic finite automaton (DFA). At every simulator step, the monitor reads physics-grounded predicates from OmniGibson/Isaac Sim, such as touching, inside, ontop, upright, dropped, and covered, then advances the automaton state. If the automaton reaches a violation state, the rollout is unsafe.

This is an important engineering choice. ManiGuard does not ask an LLM or VLM judge to watch a video and guess whether the run was safe. It also does not train a separate safety classifier. The verdict is computed from simulator state and a formal specification, which makes debugging much clearer. If an episode fails safety, you can inspect the predicate trace instead of arguing with another learned model.

System Architecture

At the system level, ManiGuard decouples the policy from the simulator. The simulator runs in the behavior conda environment with BEHAVIOR-1K, OmniGibson, and Isaac Sim. The VLA policy runs in its own stack: openpi in a JAX/openpi virtual environment, GR00T in an Isaac-GR00T/PyTorch environment, and SmolVLA in a LeRobot environment. The two sides communicate through a small contract: the evaluation client sends an observation, and the policy server returns an action chunk.

The standard observation includes one overview camera, one wrist camera, robot state, and a language prompt. The benchmark action convention is an 8-D absolute joint target for a Franka Panda: seven arm joints plus one gripper command. The eval-time controller is a JointController; the pipeline intentionally avoids end-effector IK so collection, training, and evaluation remain aligned. This detail is easy to underestimate. If the dataset uses joint actions but the eval YAML uses an EEF controller, or if training uses image_left while evaluation feeds image_right, the benchmark can silently produce misleadingly bad results.

The runtime loop is:

policy server                       ManiGuard eval client
openpi / GR00T / SmolVLA      ->    load frozen scene
receive obs                   ->    build OmniGibson env
return action chunk           <-    step JointController
                                      check task success
                                      run LTL safety monitor
                                      write results.jsonl + summary.json

The eval client records raw verdicts for every scene: whether the task succeeded, whether any safety violation occurred, whether the policy engaged the task object, and the monitor fields. Safety is contact-gated, so a policy that does nothing is not rewarded as genuinely safe behavior. It can be vacuously safe because it never touched anything, but engagement metrics reveal that it did not attempt the task. This is why the paper reports engaged-and-safe behavior and engagement-conditioned violation rate (EVR), not only aggregate violation rate.

Installing ManiGuard

You need a Linux machine with an NVIDIA GPU, compatible driver and Vulkan setup, enough disk for BEHAVIOR-1K assets, and enough GPU memory for Isaac Sim. ManiGuard simulation is not lightweight. If your policy is also large, run the simulator and policy server on separate GPUs when possible, or at least pin CUDA_VISIBLE_DEVICES.

Start from the official repository with submodules:

git clone --recursive https://github.com/NU-IDEAS-Lab/ManiGuard.git
cd ManiGuard

Install BEHAVIOR-1K from the submodule. This setup creates the behavior conda environment and installs OmniGibson, BDDL, JOYLO, dataset assets, eval tools, and primitives:

cd behavior-1k
./setup.sh --new-env --omnigibson --bddl --joylo --dataset --eval --primitives
cd ..

Install ManiGuard in editable mode:

conda activate behavior
pip install -e .
pip install -e ".[serve]"

You also need ManiGuard's custom robot asset: a Franka Panda with extended fin-ray fingers. If this asset is missing, OmniGibson may still load the stock Franka without crashing, but policies trained on ManiGuard data will approach objects and fail to grasp correctly. The expected path is:

behavior-1k/datasets/omnigibson-robot-assets/models/franka/franka_panda_longfinger/

Download it with the Hugging Face CLI:

hf download IDEAS-Lab-Northwestern/franka-panda-longfinger --repo-type dataset \
  --local-dir behavior-1k/datasets/omnigibson-robot-assets/models/franka/franka_panda_longfinger

Benchmark scenes can be snapshot-downloaded automatically from IDEAS-Lab-Northwestern/ManiGuard-Bench, or you can download them once:

hf download IDEAS-Lab-Northwestern/ManiGuard-Bench --repo-type dataset \
  --local-dir outputs/lerobot_datasets/maniguard-bench

For a headless server, set:

export OMNIGIBSON_HEADLESS=1
export CUDA_VISIBLE_DEVICES=0

If you hit Vulkan ERROR_INCOMPATIBLE_DRIVER, check VK_ICD_FILENAMES. If you hit PhysX CUDA errors or out-of-memory failures, reduce camera resolution in the eval config or move the policy server to another GPU.

Preparing Data for openpi, GR00T, and SmolVLA

ManiGuard uses a shared schema so the same dataset can feed different VLA families. Each sample stores:

state    (8,)          absolute joint config: joint_0..6 + gripper
actions  (8,)          next-step absolute joint target + gripper
image_*  256x256x3     overview cameras + wrist camera

Scripted datagen is the primary data source, producing datasets named like datagen-<fam>-v1-joint-5cam. The original dataset includes several cameras: image_left_shoulder, image_left, image_opposite, image_right, and wrist_image. For fair benchmark comparison, the evaluated policies usually consume only two cameras: one overview selected through external_cam, plus the wrist view.

For openpi, ManiGuard does not patch the upstream openpi codebase. You clone ManiGuard and openpi side by side, then ManiGuard registers train configs at runtime through maniguard/openpi_sft. The source dataset stores absolute joint targets. The openpi config converts the seven arm joints into delta actions internally, then reconstructs absolute joint targets at inference. Pi0.5 uses an action horizon of 16; pi0 uses horizon 50 and warm-starts from pi0_base. A minimal launch looks like this:

export OPENPI_ROOT=/abs/path/to/openpi
export HF_TOKEN=hf_...
export WANDB_API_KEY=...

cd ManiGuard
tools/openpi_sft/run_sft.sh \
  --config pi05-base_datagen_v1_dusty_joint_2cam_lora \
  --norm-stats

For GR00T N1.6, ManiGuard declares the simulated Franka as NEW_EMBODIMENT, still in joint space, still with 8-D state/action and one overview camera plus wrist. Training uses PyTorch/HF Trainer and component-freeze rather than LoRA. The tools are under tools/gr00t_sft/: prepare_dataset.py, bake_stats.py, run_sft.sh, run_all.sh, and push_to_hf.py. A real run prepares a dataset view, trains for roughly two epochs, and pushes a checkpoint to Hugging Face with a name like gr00t-n16-datagen-v1-<fam>-joint-2cam.

For SmolVLA, LeRobot's lerobot-train discovers features by standard key prefixes, so ManiGuard needs a prep step to rename fields: image_<external_cam> becomes observation.images.top, wrist_image becomes observation.images.wrist, state becomes observation.state, and actions becomes action. SmolVLA starts from lerobot/smolvla_base, with a SmolVLM2 backbone and a flow-matching action expert. The vision encoder is frozen and the action expert is trained. The end-to-end launcher is:

export HF_TOKEN=...
export WANDB_API_KEY=...

bash tools/smolvla_sft/run_all.sh --family clutter
# or:
bash tools/smolvla_sft/run_all.sh --all

The key practical rule is consistency. external_cam, controller preset, action dimension, and state mode must match between training and evaluation. If you train on image_left and evaluate with image_right, your policy is seeing an out-of-distribution camera view before the benchmark even starts.

Running Inference and Benchmark Evaluation

The evaluation workflow has four steps: download scenes, serve a checkpoint, run a family, and summarize results.

First, start the policy server in the model family's own environment:

# openpi / pi0.5 / pi0
python -m maniguard.serve.openpi_native \
  --config <train-config> \
  --checkpoint <ckpt-dir>

# GR00T N1.6
python -m maniguard.serve.gr00t_native \
  --checkpoint <ckpt-dir>

# SmolVLA
python -m maniguard.serve.smolvla_native \
  --checkpoint <ckpt-dir>

The servers expose an openpi-compatible websocket on 0.0.0.0:8000 by default. Then, in the behavior environment, run one family:

bash scripts/eval_family.sh jar_transport

This script runs the in-distribution level and all four OOD levels. Results are written under:

outputs/eval_logs/jar_transport_joint/
  ID/results.jsonl
  ID/summary.json
  OOD/target/results.jsonl
  OOD/language/results.jsonl
  OOD/location/results.jsonl
  OOD/env/results.jsonl

To run all six families, restart or swap the matching checkpoint for each family and run:

for fam in clutter_pickup cabinet_pickup stack_retrieve jar_transport lid_transport dusty_transfer; do
  bash scripts/eval_family.sh "$fam"
done

Summarize the logs with:

python tools/eval_summary.py outputs/eval_logs/*_joint --full

Read these metrics together:

Main Results from the Paper

The paper evaluates zero-shot and fine-tuned VLAs across more than 23,000 rollouts. The first result is that task success is not enough: 6-21% of successful rollouts still violate the specification. In other words, a success-only dashboard can make a policy look deployable even when it is physically unsafe during execution.

The second result is that fine-tuning on the ManiGuard data suite helps substantially, but does not solve safety. Safe task completion rises from near zero to roughly 7.5-29.8%. Engaged-and-safe behavior rises from 16-40% to 51-72%. However, among engaged rollouts, 21-42% still violate the safety specification. Two of the six task families remain below 2% safe success for every evaluated policy. That is a strong signal that "more demonstrations of the same kind" is not enough; the field needs better safety-structured data, better monitors, or different policy objectives.

The third result is that OOD perturbations are dangerous in contact-rich manipulation. ManiGuard holds the safety specification fixed while changing target appearance, language, object location, or environment. That measures the property we actually care about: does the policy preserve the same safety rule under a small distribution shift? If EVR jumps when the object moves, the policy may have learned trajectory shortcuts rather than the safety principle.

Beginner Checklist

If you are new to the stack, do not start by running the full 1,000-scenario benchmark. Use a small path first:

1. Install the behavior environment and confirm OmniGibson can load one scene.
2. Download the long-finger Franka asset and verify the exact directory.
3. Download ManiGuard-Bench or let the eval client pull it from Hugging Face.
4. Pick one readable family such as jar_transport.
5. Serve a released Hugging Face checkpoint before training your own.
6. Run LEVELS="base" REPEAT=1 bash scripts/eval_family.sh jar_transport as a smoke test.
7. Open results.jsonl, find an unsafe-success rollout, and inspect the monitor fields.
8. Only after you understand the summary should you expand to OOD levels and multiple seeds.

For a product team, ManiGuard should sit after unit tests and controller-level safety checks, but before real robot deployment. It does not replace emergency stops, joint limits, collision checking, or runtime action guards such as SafeContract. What it gives you is a sharper evaluation question: which model completes the task while preserving the safety specification?

When to Use ManiGuard-Bench

Use ManiGuard-Bench when you need to compare openpi, GR00T, and SmolVLA on the same task family, camera convention, action controller, and safety specification. Use it when you suspect a fine-tune improved task success but made the policy more reckless. Use it when you need to know which OOD axis makes a policy unsafe.

Do not treat ManiGuard as proof that a robot is absolutely safe. The benchmark runs in OmniGibson/Isaac Sim and partially on a real Franka setup, but your factory cell, gripper, camera mounting, object meshes, and human workflow can introduce new risks. The right use is as a safety regression suite: every time you change a checkpoint, dataset, prompt template, or camera mapping, rerun the same frozen scenarios and compare safe success, unsafe success, engagement, and EVR.

Related Posts

• SmolVLA on LeRobot
• GR00T N1.6 fine-tuning
• SafeContract for VLA monitoring