Fine-Tune DM0.5/OpenDM on SO101

DM0.5 is one of the more useful open-source VLA releases for robotics developers because it does not stop at a model checkpoint. Dexmal also released OpenDM: training scripts, dataset registration examples, LoRA recipes, benchmark workflows, and an HTTP inference service. For someone learning robot manipulation, this matters more than a headline success rate. You can trace the whole path from JSONL data to normalization statistics, from LoRA adapters to merged inference, and from image keys to action chunks.

This guide focuses on one concrete workflow: fine-tuning DM0.5/OpenDM on the SO101 Pick Cube task with LoRA, then using RoboTwin2.0 to understand how DM0.5 is evaluated on a larger manipulation benchmark. SO101 is a low-cost robot arm in the LeRobot ecosystem. RoboTwin2.0 is a much larger bimanual simulation benchmark and data generator. They are not the same problem, but they teach complementary lessons: SO101 gives you a small real-robot pipeline you can reproduce; RoboTwin2.0 shows what robust manipulation evaluation looks like when you scale tasks, objects, embodiments, and domain randomization.

If you already know the basics from VLA and LeRobot workflows, treat this article as the next step. We will look at data schema, action dimension, action mode, chunk size, LoRA target modules, training commands, inference commands, and the common failure modes that cause a model to "train" but fail on the robot.

What DM0.5 and OpenDM provide

The official Hugging Face model card describes DM0.5 as Dexmal's next-generation Vision-Language-Action model for open-world robot control. It builds on the embodied-native approach introduced by DM0: instead of treating robot actions as a small downstream afterthought, the framework trains with embodied sensorimotor data, visual data, language instructions, manipulation trajectories, navigation data, and embodied reasoning tasks.

The DM0 paper frames the system as a three-stage pipeline: pretraining, mid-training, and post-training. The important idea for roboticists is that the model learns from mixed supervision. Some samples are vision-language conversations, some are embodied reasoning examples, and some are robot trajectory samples. For robot data, each timestep can contain multi-view images, a natural language instruction, proprioceptive state, and an action target. The training collator then builds action chunks so the model predicts a short future trajectory rather than one isolated motor command.

OpenDM exposes this idea in a developer-friendly way. The current public model family includes:

OpenDM reports strong benchmark numbers: 99.0 average success on LIBERO and 93.5 average on RoboTwin2.0, with 93.6 Clean and 93.3 Randomized. For SO101, the key release is the practical one: a Pick Cube checkpoint and a LoRA SFT workflow that lets you adapt the base model to a small real robot without full fine-tuning.

The paper idea: embodied-native VLA

A simple way to build a VLA is to take a pretrained VLM, add an action head, and fine-tune on robot demonstrations. That approach is understandable, but it has three recurring problems:

1. The model understands objects and language, but it may not have strong physical priors.
2. Robot data is scarce, so the policy overfits to camera placement, background, object poses, and demonstration speed.
3. Action prediction can be disconnected from temporal reasoning: the model sees "pick the cube" but may not know whether the task is at approach, grasp, lift, transport, or release.

DM0 and DM0.5 address this by putting embodied data deeper into training. The DM0 paper describes unified pretraining over vision-language data, driving data, and embodied corpora so the model learns semantic knowledge and physical priors together. The mid-training stage adds embodied reasoning tasks such as task decomposition, subtask prediction, action QA, temporal reasoning, and task progress estimation. The model is therefore asked not only "what is in this image?" but also "what part of the task is happening?" and "what action should bridge these frames?"

For beginners, the practical takeaway is simple: a useful VLA must do more than recognize an orange cube. It must know whether the gripper is open, whether the cube has already been lifted, whether the wrist camera is partially occluded, and whether the next chunk should approach, close, lift, or place. LoRA fine-tuning on SO101 adapts that broad capability to the specific state/action space of a six-degree robot arm.

Architecture in practice: backbone, Action Expert, chunks

You can think of DM0.5 in OpenDM as three working layers:

Action chunking is the detail most beginners miss. The model is not always asked to output a single next action. In the OpenDM SO101 guide, the chunk size is 50, the action dimension is 6, and the action mode is relative. In the RoboTwin2.0 guide, the chunk size is also 50, but the action dimension is 14 because the ALOHA RoboTwin2 embodiment has two arms, each with six joints and one gripper value.

That gives you a strict rule: training, inference, and evaluation must use the same action dimension, action mode, camera order, and chunk size. If you train SO101 with relative actions but interpret the output as absolute joint positions at inference time, the robot can move in the wrong direction. If images_1 is the top camera during training and you send the wrist camera as images_1 during deployment, the model receives a distribution shift you created yourself.

Why LoRA is the right first adaptation method

Full fine-tuning a 6B-parameter VLA is expensive and risky. You need large GPUs, long training runs, and careful learning rates to avoid degrading the base model. LoRA is a better first tool: it freezes most of the base model and trains small low-rank adapters inserted into linear layers. In OpenDM's SO101 recipe, LoRA is enabled with use_lora=True, target modules are all-linear, and AdamW optimizes the LoRA and selected trainable weights.

The official SO101 Pick Cube recipe is:

This is exactly the kind of task where LoRA makes sense. Pick Cube is much narrower than the base model's training distribution. You do not want to rewrite the model's general visual-language knowledge; you want it to learn how SO101's top/wrist images and six-dimensional state map to six-dimensional relative actions.

Install OpenDM

OpenDM recommends Docker because it avoids CUDA, PyTorch, and flash-attn version mismatches. Use Ubuntu 20.04/22.04, NVIDIA drivers, Docker, and NVIDIA Container Toolkit. Reference training uses multiple GPUs, but a one-GPU smoke test is still useful for checking the data loader, model load, and checkpoint output.

git clone https://github.com/dexmal/opendm.git
cd opendm

docker run -it --rm --gpus all --network host \
  --name opendm \
  --shm-size=16g \
  -v "$PWD":/app/opendm \
  -w /app/opendm \
  dexmal/opendm:latest /bin/bash

conda activate opendm
pip install -e .

For a local install, use Python 3.10:

conda create -n opendm python=3.10 -y
conda activate opendm

pip install torch torchvision \
  --index-url https://download.pytorch.org/whl/cu128

pip install ninja packaging
MAX_JOBS=2 pip install flash-attn --no-build-isolation

pip install -e .

If flash-attn fails to build, do not spend the whole day debugging your host environment before you understand the workflow. Start with Docker, confirm the training path works, then come back to local installation later if you really need it.

Prepare the SO101 Pick Cube data

The SO101 guide uses the Dexmal/so101_pick_cube dataset. The official runner downloads it from Hugging Face, extracts archive parts when needed, and organizes it under ./data/so101_pick_cube.

pip install -U huggingface_hub
script/so101_runner.sh dataset

If your data lives elsewhere:

script/so101_runner.sh dataset --data-root /path/to/data/so101_pick_cube

The expected layout is:

data/so101_pick_cube/
  jsonl/
    episode_00000.jsonl
    episode_00001.jsonl
    ...
  videos/
    so101_YYYYMMDD_HHMMSS_filtered/
      file-000.mp4_top.mp4
      file-000.mp4_wrist.mp4
      ...

Each JSONL row represents one frame:

{
  "prompt": "Pick the cube and place it in the plate.",
  "state": [-2.29, -102.81, 95.82, 54.02, 2.68, 0.68],
  "action": [-0.84, -104.22, 99.16, 54.24, 2.24, 0.16],
  "is_robot": true,
  "extra": {
    "subtask": "Pick the cube and place it in the plate.",
    "timestamp": 0.70,
    "episode_index": 0,
    "cube_color": "orange"
  },
  "images_1": {
    "type": "video",
    "url": "episode_00000/camera_top.mp4",
    "frame_idx": 21,
    "_camera_name": "top"
  },
  "images_2": {
    "type": "video",
    "url": "episode_00000/camera_wrist.mp4",
    "frame_idx": 21,
    "_camera_name": "wrist"
  }
}

The fields to audit first are:

OpenDM computes normalization statistics automatically when the matching file is missing under ./norm_stats/. Recompute those stats whenever you change the data distribution, action mode, action dimension, or chunk size. Reusing old stats with new data is a quiet failure mode: training still runs, but the action scale is wrong.

Run LoRA training on SO101

Download or mount the base checkpoint first:

huggingface-cli download Dexmal/DM05 \
  --local-dir ./checkpoints/DM05

Then run the official training command:

script/dm05_launcher.sh \
  --exp playground/dm05_so101_lora.py \
  --task train \
  --nproc_per_node 8 \
  --model-config.model-name-or-path ./checkpoints/DM05

playground/dm05_so101_lora.py already sets the dataset name, action mode, chunk size, attention settings, learning rate, warmup, batch size, save interval, and training steps. When you are starting out, resist the urge to override everything. Run the reference recipe first and get a known baseline.

For a one-GPU smoke test:

script/dm05_launcher.sh \
  --exp playground/dm05_so101_lora.py \
  --task train \
  --nproc_per_node 1 \
  --model-config.model-name-or-path ./checkpoints/DM05 \
  --trainer-config.num-train-steps 200 \
  --trainer-config.save-steps 100

Do not judge policy quality from a smoke test. It only answers these questions:

1. Can the environment load the model?
2. Is the dataset path correct?
3. Can the loader decode the video frames?
4. Does the trainer write checkpoints to the expected directory?

After training starts, inspect the trainable summary:

user_checkpoints/<output_dir>/trainable_summaries/
  dm05_lora_so101_pick_cube.json

The summary should show that target_modules resolves from all-linear, unexpected_trainable_parameters is empty, and the dense saved modules include action projections, time MLPs, and time modulators. If unrelated parameters are trainable, you are no longer running the intended LoRA recipe.

Run inference with an SO101 checkpoint

For the published checkpoint:

huggingface-cli download Dexmal/DM05-SO101-Pick-Cube \
  --local-dir ./checkpoints/DM05-SO101-Pick-Cube

Start inference:

script/dm05_launcher.sh \
  --exp playground/dm05_so101_lora.py \
  --task inference \
  --nproc_per_node 1 \
  --model-config.model-name-or-path ./checkpoints/DM05-SO101-Pick-Cube \
  --inference-config.output-action-dim 6

For your own LoRA run, point to a step checkpoint:

script/dm05_launcher.sh \
  --exp playground/dm05_so101_lora.py \
  --task inference \
  --nproc_per_node 1 \
  --model-config.model-name-or-path ${TRAINING_OUTPUT_DIR}/checkpoint-4000 \
  --inference-config.output-action-dim 6

The loader reads adapter_config.json, finds the recorded base model, and merges the adapter for inference. That is convenient, but it means your checkpoint artifact must be complete: adapter files, config files, normalization statistics when applicable, and the metadata needed to locate the base model.

OpenDM's inference service exposes /process_frame and accepts multipart/form-data: task text, state array, and images matching the configured image keys. For SO101, the state must be six-dimensional and the camera order must match training. Before real-robot closed-loop control, add a safety wrapper:

Why RoboTwin2.0 belongs in this discussion

RoboTwin2.0 is not the SO101 dataset. It is a bimanual simulation data generator and benchmark with 50 tasks, 731 object instances across 147 categories, and structured domain randomization over clutter, lighting, background, tabletop height, and language. The paper reports more than 100,000 expert trajectories across tasks and embodiments.

So why should an SO101 user care? Because DM0.5 uses RoboTwin2.0 to demonstrate manipulation fine-tuning and robust evaluation at a larger scale. OpenDM's DM05-robotwin2 guide reports:

The important signal is that Randomized is close to Clean. The model is not only memorizing a clean tabletop layout. It remains strong under environment variation. When you collect SO101 data, the same principle applies: randomize cube positions, lighting, cube colors, modest camera shifts, background, and instruction phrasing. If all your demonstrations use the same table corner and one camera angle, LoRA will learn that visual shortcut instead of learning a robust pick-and-place behavior.

Training RoboTwin2.0 in OpenDM

OpenDM registers the dataset as robotwin2_generalist in opendm/dataset/robotwin2.py. The key settings are:

Prepare the data:

mkdir -p data/.hf_downloads/robotwin
hf download Dexmal/robotwin2-full \
  --repo-type dataset \
  --local-dir data/.hf_downloads/robotwin

cat data/.hf_downloads/robotwin/robotwin2.tar.part-* \
  | tar -xf - -C data

hf download Dexmal/DM05 --local-dir checkpoints/DM05

Run training:

script/dm05_launcher.sh \
  --exp playground/dm05_robotwin2.py \
  --task train \
  --nproc_per_node 8 \
  --data-config.dataset-name robotwin2_generalist \
  --model-config.model-name-or-path ./checkpoints/DM05 \
  --model-config.chunk-size 50 \
  --trainer-config.num-train-steps 100000

Run inference:

script/dm05_launcher.sh \
  --exp playground/dm05_robotwin2.py \
  --task inference \
  --nproc_per_node 1 \
  --model-config.model-name-or-path ./checkpoints/DM05-robotwin2-bf16 \
  --model-config.chunk-size 50 \
  --inference-config.output-action-dim 14 \
  --inference-config.image-keys images_1 images_2 images_3 \
  --inference-config.port 7891

For benchmark evaluation, keep action_horizon: 50, action_mode: absolute, and camera order head_camera_rgb,left_camera_rgb,right_camera_rgb. The benchmark runner evaluates 100 episodes for a chosen task and setting. To reproduce the aggregate Clean and Randomized scores, evaluate all 50 tasks under both settings and aggregate the success rates. A single task rollout is useful for debugging, not for a benchmark claim.

Beginner checklist before spending GPU hours

Before running a long job, go through this checklist:

A practical habit: before running closed-loop on the robot, log 20 inference requests with the same image/state and inspect the returned action chunk. If the action changes aggressively when the observation is unchanged, check normalization, checkpoint loading, image preprocessing, and camera ordering before blaming the model.

Results and expectations

The official OpenDM numbers are strong on simulation benchmarks, especially LIBERO and RoboTwin2.0. The SO101 release is better interpreted as a reproducible real-robot adaptation path: you get a base model, a Pick Cube dataset, a LoRA recipe, and a published checkpoint. Your own result will depend heavily on camera stability, lighting, cube placement distribution, calibration, and how closely your real setup matches the dataset.

For a first real-robot experiment, do not aim for a polished demo immediately. Aim for a controlled evaluation table:

This turns a subjective demo into a policy debugging loop. If fixed-pose success is high but randomized-pose success is low, collect more varied demonstrations. If top-camera changes break the policy, add camera variation or stabilize your mount. If the gripper closes too early, inspect temporal alignment and action chunks around the approach-to-grasp transition.

Conclusion

SO101 plus DM0.5/OpenDM is a practical entry point into open-source VLA manipulation. The robot is small enough to learn on, the Pick Cube task is clear enough to debug, and the model stack is modern enough to teach the real workflow: structured robot data, normalization, LoRA adapters, action chunks, and inference services. RoboTwin2.0 adds the larger lesson: robust manipulation policies come from variation, evaluation discipline, and consistent action/camera semantics, not from one successful video.

If you remember only three rules, remember these: your data schema must match the registry, your normalization statistics must travel with the checkpoint, and training/inference/evaluation must agree on action mode and chunk size. Most painful VLA failures are violations of one of those rules.

References

• OpenDM GitHub repo
• DM05 model card on Hugging Face
• DM05 SO101 LoRA Training Guide
• DM05 RoboTwin2.0 Training and Evaluation Guide
• DM0 paper: Embodied-Native VLA
• RoboTwin2.0 project
• RoboTwin2.0 paper
• LeRobot SO101 docs

Related Posts

• PEFT/LoRA Fine-tuning and VLA Deployment
• RoboTwin 2.0 dual-arm manipulation
• SO101 Isaac Lab LeRobot sim2real