DexVerse: Benchmark OpenVLA and π0.5

DexVerse is a new benchmark for dexterous manipulation: robots do not just pick and place with a parallel gripper; they control an arm plus a multi-finger hand to open laptops, squeeze scissors, pour from containers, lift trays with two hands, push objects through contact, and complete long-horizon procedures such as making coffee. The interesting part is that DexVerse is not a small demo suite. The paper introduces 100 tasks, 3 robot arms, 6 dexterous hands, controllable visual variation, VR teleoperation, demonstration data, and baseline evaluations for OpenVLA, π0.5, DP3, and Diffusion Policy.

If you have already read our OpenVLA deep dive, π0 overview, or the guide to EXPO-FT for π0.5, DexVerse is the missing stress test. It asks whether modern VLA policies can handle dexterous hands, contact-rich motion, embodiment changes, and long-horizon task structure, not just clean tabletop gripper tasks.

This article is based on the DexVerse: A Modular Benchmark for Multi-Task, Multi-Embodiment Dexterous Manipulation arXiv paper and the official ycyao216/DexVerse repository. As of this writing, the repo has released the task suite and asset download pipeline. The full baseline code, complete teleoperation/data-collection tooling, full Shadow Hand demonstration dataset, and cross-embodiment assets are still listed in the roadmap or marked as coming soon. The guide below therefore separates what you can run from the current repository from the training and inference protocol reported in the paper.

What problem does DexVerse solve?

Popular robot-learning benchmarks such as LIBERO, CALVIN, Meta-World, and many LeRobot-style setups are useful for imitation learning, but many of their tasks center on simple grippers. With a gripper, a policy can often learn a repeated pattern: reach, grasp, move, release. A dexterous hand is a different control problem. Pouring from a can requires the correct contact region, enough tilt, stable lift, and coordinated finger joints. InsertPen needs precise alignment. PushT depends on friction and stable contact. OpenStapler requires understanding an articulated object.

DexVerse evaluates three kinds of generalization at the same time. The first is cross-task generalization: can a method handle primitive, functional, articulation, contact-rich, bimanual, multi-goal, and long-horizon tasks? The second is cross-embodiment generalization: are tasks hard-coded to one hand, or can the same task definition support multiple arm-hand pairs? The third is visuomotor robustness: can a policy survive changes in texture, lighting, HDR background, exposure, and camera viewpoint?

The paper makes a useful point for VLA practitioners: internet-scale pretraining does not automatically become dexterous manipulation competence. On the baseline split, π0.5 and DP3 both reach 0.34 mean success, Diffusion Policy reaches 0.32, and OpenVLA reaches 0.19. This is not an argument that VLA is useless. It is evidence that visual-language priors alone do not solve high-DoF multifinger control, force timing, and sub-centimeter contact alignment.

How are the 100 tasks organized?

DexVerse groups its 100 tasks into 8 categories:

Each task is specified with a scene, assets, robot embodiment, observation and action interfaces, initialization distribution, success conditions, and randomization settings. For a beginner, the key idea is simple: a task is not just a Python file that draws objects. It is a complete simulator contract. It tells the system how to reset, what the robot observes, what action space the policy controls, and what counts as success.

For example, opening a drawer can be scored by a prismatic joint crossing a threshold. Pouring from a can can require both lift and tilt. A multi-goal task can require the auxiliary object to be lifted while the primary object’s articulated joint also reaches its target. This matters because policy evaluation should not be subjective. You want a reproducible predicate that can be applied across 50 closed-loop rollouts.

Environment architecture: config-driven Isaac Lab

DexVerse is built on NVIDIA Isaac Sim 5.1.0 and Isaac Lab v2.3.2. The repository is organized as an Isaac Lab extension:

DexVerse/
├── source/dexverse/
│   ├── dexverse/
│   │   ├── tasks/
│   │   ├── assets/
│   │   ├── devices/
│   │   ├── robot_agents/
│   │   └── utils/
│   ├── demonstrations/
│   ├── docker_utils/
│   └── docs/
└── scripts/
    ├── list_envs.py
    ├── zero_agent.py
    ├── random_agent.py
    ├── teleop_agent.py
    ├── record_demos.py
    └── run_dexverse.py

Isaac Lab’s manager-based environment interface lets observations, actions, events, terminations, and reward terms be specified through configuration classes. DexVerse uses that pattern to decouple task logic from robot embodiment logic. A task family keeps shared reset, asset, and success logic; each task overrides its object, target, and threshold. A robot config defines the arm, hand, initial pose, controller, and action interface. In principle, this lets the same task run with Shadow Hand, Allegro Hand, LEAP Hand, or another configured hand without rewriting the task itself.

The paper lists 3 arms: Franka Research 3, UR10e, and xArm 7. It also lists 6 hands: Sharpa Wave, WUJI Hand, Shadow Hand, Inspire Hand, Allegro Hand, and LEAP Hand. The current README notes that the initial asset release currently includes Shadow Hand, while the remaining robot combinations are coming later. That distinction is important: the paper describes the intended benchmark scope; the repository release is still being staged.

Dataset: 3,180 VR teleoperation demonstrations

DexVerse uses Apple Vision Pro through Isaac Lab’s CloudXR-based XR teleoperation interface. The human operator sees simulation feedback in the headset. Human wrist pose becomes the target pose for the robot end effector, the arm follows it through inverse kinematics, and human hand motion is converted into dexterous-hand joint targets through optimization-based dex-retargeting.

The paper reports 3,180 demonstration trajectories. The counting is straightforward: 56 single-goal tasks receive 55 demonstrations each; 50 are collected with Shadow Hand and one is collected with each of five additional hand embodiments. The 5 long-horizon tasks receive 20 demonstrations each. Each demonstration is stored as action-state pairs from teleoperation. During replay, the simulator restores recorded states and regenerates requested observation terms locally, such as RGB, depth, point cloud, proprioception, or simulator state.

This state-replay design is practical. If you only store videos or actions and replay physics from the beginning, small numerical differences can accumulate across machines because of solver details, hardware, and floating-point behavior. Restoring recorded simulator states reduces rollout drift and keeps the dataset portable. It also lets researchers modify observation presets later without duplicating every RGB/depth frame in advance.

Installing DexVerse from the current repository

This section follows the official README. You need a Linux machine with an NVIDIA GPU capable of running Isaac Sim. DexVerse is not a lightweight laptop benchmark. If you start on unsupported hardware, you are likely to spend more time debugging Isaac dependencies than learning the benchmark.

Create a workspace where Isaac Lab and DexVerse sit side by side:

mkdir -p ~/workspace/dexverse-lab
cd ~/workspace/dexverse-lab
conda create -n dexverse python=3.11
conda activate dexverse
pip install --upgrade pip

Install Isaac Sim 5.1.0 and the CUDA 12.8 PyTorch builds recommended by the README:

pip install "isaacsim[all,extscache]==5.1.0" --extra-index-url https://pypi.nvidia.com
pip install -U torch==2.7.0 torchvision==0.22.0 torchaudio==2.7.0 --index-url https://download.pytorch.org/whl/cu128
pip install "setuptools==65.0.0"
pip install "flatdict==4.0.1" --no-build-isolation

Install Isaac Lab v2.3.2:

git clone https://github.com/isaac-sim/IsaacLab.git --branch v2.3.2
cd IsaacLab
sudo apt install cmake build-essential
./isaaclab.sh --install

Then install DexVerse as an editable extension:

cd ~/workspace/dexverse-lab/DexVerse
python -m pip install -e source/dexverse

Robot and object assets are not stored directly in the git repository. They are hosted on the gated Hugging Face dataset dexverse/DexVerse_release, so you must log in and accept the dataset terms:

pip install huggingface_hub
hf auth login
python scripts/asset_tools/download_robot_agents.py --all
python scripts/asset_tools/download_assets.py --all

The README says download_assets.py --all pulls several gigabytes: core assets around 410 MB, the ManiTwin object pool around 2.2 GB, HDRIs around 1.8 GB, plus long-horizon meshes. If you only want a small smoke test, use --help or download_robot_agents.py --list to download a smaller subset.

Smoke test: list environments and run dummy agents

After installation, start with:

python scripts/list_envs.py

This confirms that Isaac Lab can see the extension and that environments are registered. Then pick a simple task from the output, for example Dexverse-Relocate-v0, and run the zero and random agents:

python scripts/zero_agent.py --task=Dexverse-Relocate-v0
python scripts/random_agent.py --task=Dexverse-Relocate-v0 --num_envs=4

The zero agent sends zero actions at every step. The random agent samples random actions. Neither is supposed to solve the task. They test whether the scene loads, asset paths are correct, the simulator steps, and the action space is wired correctly. In simulation-heavy robotics projects, this kind of smoke test is essential because many failures come from asset paths, USD/URDF issues, GPU drivers, Isaac extension registration, or dependency conflicts rather than the learning algorithm.

Training OpenVLA and π0.5 using the paper protocol

The paper evaluates four policy families on a 19-task baseline split. All four are trained on the same 950 episodes: 19 tasks times 50 episodes per task from the teleoperation corpus. Evaluation is closed-loop in the same simulator, with 50 rollouts per task and identical termination criteria. That is the minimum protocol you should preserve if you want comparable numbers.

For OpenVLA, the paper fine-tunes openvla-7b using the OFT recipe. It replaces the original discrete 7-token action head with a continuous L1 regression head that emits an 8-step chunk of absolute joint targets. In practical terms, instead of asking the VLA to generate discrete action tokens, the action head learns continuous joint target vectors for the dexterous arm-hand system. That change is necessary because a multi-finger hand has a much denser action space than a simple gripper.

For π0.5, the paper treats it as a VLA transformer with a flow-matching action expert. π0.5 reaches 0.34 mean success, matching DP3 and beating OpenVLA in aggregate. But the result should not be read as “π0.5 solves dexterity.” PushT is still 0.00, InsertPen is 0.06, and OpenLaptop is 0.04. Even strong VLA models still lack robust force feedback, contact correction, and stable alignment under tight tolerances.

Once the official baseline code and dataset are fully released, a realistic workflow will look like this:

# Pseudocode: script names may differ in the official baseline release
python scripts/demo_tools/prepare_dataset.py \
  --tasks baseline_19 \
  --episodes-per-task 50 \
  --obs rgb,state \
  --action absolute_joint_targets \
  --out data/dexverse_baseline_19

python scripts/train_openvla_oft.py \
  --model openvla-7b \
  --dataset data/dexverse_baseline_19 \
  --action-head continuous_l1_chunk \
  --chunk-size 8

python scripts/train_pi05.py \
  --dataset data/dexverse_baseline_19 \
  --obs rgb,state \
  --action absolute_joint_targets

Do not treat the snippet above as official DexVerse commands. It is a skeleton to make the training components clear: choose a task split, convert demonstrations, select observation terms, choose an action representation, fine-tune the action head, and save a checkpoint. When the authors release their baseline scripts, use their commands first.

Inference and closed-loop evaluation

Inference in DexVerse is not like text-model inference. At each time step, the policy receives an observation, emits an action or action chunk, the simulator advances, and the policy observes the next state. For VLA policies, the observation usually includes RGB, proprioceptive state, and a task instruction. For DP3, it is point cloud plus proprioception. For state-based Diffusion Policy, it is proprioceptive state.

The evaluation loop should preserve the benchmark rules:

for task in baseline_19:
  for seed in 50_rollouts:
    env.reset(task, seed)
    while not done:
      obs = env.get_observation()
      action = policy(obs, instruction)
      env.step(action)
    record_success(env.success)

Log at least the task name, checkpoint, observation mode, action mode, seed, success/failure, episode length, termination reason, and rollout video. For contact-rich tasks, mean success is not enough. A policy can fail because it grasps the wrong affordance, misaligns the wrist, touches a forbidden region, launches the object out of bounds, or loses timing during contact. Termination reason and video make those failures debuggable.

Reading the benchmark results

The reported mean success rates are:

Some tasks are much easier than others. On BimanualLiftCarton, π0.5 reaches 1.00, Diffusion Policy reaches 0.94, DP3 reaches 0.90, and OpenVLA reaches 0.60. On RetrieveCup, all four are weak: 0.02, 0.02, 0.06, and 0.04. On articulation, OpenStapler is strong for all methods and OpenVLA reaches 0.92, but SlideUtilityKnife is 0.00 for everyone. On tool use, DP3 is strong on FunctionalPourMug with 0.64, while Diffusion Policy is strongest on FunctionalPourCan with 0.38. On precision, PushT is 0.00 across all methods.

The lesson is that dexterous manipulation contains many micro-regimes. Winning on lifting does not imply winning on tool use. RGB plus state may be enough for some pick-and-lift tasks. Point clouds can help localize tool geometry. Language conditioning and flow-matching action experts can help with articulated and precision-contact tasks. But no representation dominates everything.

For a production robotics team, this is a useful warning. If your real task is contact-rich insertion, do not demonstrate a VLA on pick-and-place and assume the model is ready. Build an evaluation split with explicit success predicates, enough seeds, and failure-mode logging for contact and alignment.

Where should beginners start?

If you are new to DexVerse, use this order:

1. Run the repository with zero and random agents to understand the Isaac Lab extension and asset pipeline.
2. Read source/dexverse/docs/envdocs.md to understand task intent and success conditions.
3. Pick one primitive task, one articulation task, and one functional task; compare their configs.
4. When the demonstration data is fully released, convert one small task first instead of starting with all 19 baseline tasks.
5. Train a small baseline before fine-tuning a 7B VLA.
6. When running VLA inference, log videos and termination reasons because a low success rate without rollout evidence is almost impossible to debug.

If your main stack is LeRobot/OpenVLA, DexVerse is best viewed as a high-end dexterous manipulation benchmark rather than your first robot-learning tutorial. It is the benchmark you reach for when you want to know whether a policy has escaped tabletop gripper bias. For neighboring context, compare it with DexOrA-VLA bimanual dexterous manipulation and RoboTwin 2.0 dual-arm manipulation.

Lab deployment checklist

Before spending serious GPU time on OpenVLA or π0.5 with DexVerse, check the following:

For a product-oriented team, I would not start with all 100 tasks. Choose five representative tasks first: one primitive sanity check, one articulation task, one functional tool-use task, one precision-contact task, and one long-horizon task. Once the pipeline is reliable, expand to the 19-task baseline split and then to the full taxonomy.

Conclusion

DexVerse matters because it brings VLA evaluation back to technical reality. OpenVLA and π0.5 are strong models, but dexterous manipulation is not just “look at an image and emit an action.” Multi-finger hands need contact timing, force-aware correction, embodiment-aware action representation, and broad evaluation. A best mean success of 0.34 is not a failure; it is a benchmark floor for the community to improve.

The main thing to watch next is the repository release of baseline code, complete teleoperation tooling, and demonstration data. Once those are stable, DexVerse can become a serious benchmark for testing VLA models, diffusion policies, and cross-embodiment dexterous control.

Related Posts

• OpenVLA deep dive
• π0 overview
• EXPO-FT for π0.5