SLIM-0.5B: Lightweight VLA on LIBERO

SLIM-0.5B is an important design point for robot manipulation. Instead of placing every part of control inside a many-billion-parameter Vision-Language-Action model, the paper SLIM-0.5B: Learning Action-Grounded Predictive Latents for Robot Manipulation builds a compact policy with about 0.47B parameters. It uses DINOv2 for visual observations, T5-small for language instructions, and a compact Mixture-of-Transformers backbone that models interactions between observation latents and continuous action tokens. The central idea is not just model compression. SLIM trains the observation latent space to understand which actions explain observed changes, and which future latent changes should follow from an action chunk.

In practical terms, SLIM asks a useful robotics question: does a manipulation policy really need a huge open-domain VLM at every control step? For tasks such as pick, place, wipe, stack, and object rearrangement in LIBERO, the robot must understand the instruction, attend to the right object, track the gripper, and predict the consequence of the next action chunk. Many open-domain language capabilities of large VLMs are not the bottleneck inside a continuous control loop. SLIM therefore shifts the center of computation from "bigger language backbone" to "action-grounded latent interaction".

If you are new to VLA policies, start with our VLA Models overview. If you want more context on datasets, action formats, and robotics training workflows, the LeRobot hands-on guide will make the SLIM recipe easier to follow.

Official Sources

The SLIM-0.5B paper appeared on arXiv on August 10, 2026. The authors are Jingkai Wang, Zihan Tang, Gu Zhang, Mingyu Cao, Jiapeng Chen, Jingjiao Zhao, Xiansheng Chen, Pengwei Wang, Lemao Liu, and Dejing Dou. The official resources are:

The repository includes a README, data documentation, environment documentation, training scripts, evaluation scripts, public checkpoints, and reproducibility notes. This article does not replace the official docs. It explains the paper in a beginner-friendly way: the idea, architecture, LIBERO data recipe, installation, training, inference, evaluation, and how to interpret the reported numbers.

What Problem Does SLIM Solve?

Modern VLA policies usually follow one of two broad paths. The first path uses a large vision-language backbone, adds action tokens or an action head, and lets the same model process images, language, and action generation. This gives the policy strong semantic priors, but it is expensive. Every control step may require a large multimodal forward pass. On a real robot, latency and GPU memory are not minor engineering details. They determine replanning frequency, how many robots can share a server, and whether the policy can run on a workstation or edge GPU.

The second path uses world models or world-action models. These methods predict future images, videos, or joint video-action trajectories to learn dynamics. The motivation is correct for robotics: actions should be grounded in the physical changes they cause. The weakness is cost. Pixel-level prediction spends capacity on details that often do not matter for the next action, such as background texture, lighting, or visual regions unrelated to contact. If the model must roll out future video during inference, the control loop becomes even heavier.

SLIM chooses a third path: predict the future in the same latent space used by the policy, not in pixel space. It learns with masked trajectory prediction:

Current observation + future observation -> reconstruct action chunk
Current observation + action chunk       -> predict future observation latent

These two directions make the representation action-grounded. A generic visual latent may know that "this is a bowl" or "this is a mug", but not which action caused a particular transition. An action-grounded latent is trained to preserve information about the object, gripper, contact region, and state variables that matter for manipulation.

SLIM-0.5B Architecture

SLIM separates the policy into four main components:

In the LIBERO setup, SLIM uses two camera views: a workspace view and a wrist view. Images are resized to 224x224. Actions are 7D, and the action horizon is 8, so the policy predicts an 8-step action chunk. At inference time, the policy uses 4 flow-sampling steps to generate the action chunk. That is one reason the policy is lightweight compared with larger VLA or diffusion-style baselines that use more sampling steps or larger control backbones.

The MoT backbone has two main streams: an observation stream and an action stream. The observation stream processes image latents and future-slot embeddings. The action stream processes continuous action tokens; SLIM does not discretize actions into text vocabulary items. The language embedding acts as task-level conditioning. It is not the place where all control computation happens.

A common misunderstanding is to assume that "predictive latent" means the deployed robot must generate future images or future observations before acting. That is not how SLIM is used. Future latents are used during Stage 1 to shape the representation. During Stage 2 and inference, the policy receives the current observation, proprioception, language embedding, learned future-slot embeddings, and a noised action chunk for flow matching. It does not require a real future observation and does not render a future video during deployment.

Action-Grounded Predictive Latents

Consider a short LIBERO trajectory. The instruction asks the robot to place a mug to the right. The gripper approaches the mug, closes, lifts, and moves the mug. A normal visual encoder can store many things: table color, background texture, camera pose, distractor objects, and lighting. A manipulation policy needs the parts related to action: where the mug is, where the gripper is, whether contact has happened, and how an action chunk changes the scene.

SLIM uses two complementary objectives:

IDM asks: "Given the change from current latent state to future latent state, which action explains it?" FDM asks the reverse: "Given this current observation and action chunk, which future latent factors should change?" Training both objectives pushes the latent space toward action-observation interaction rather than generic image recognition.

The Stage 1 loss is:

L_stage1 = lambda_IDM * L_IDM + lambda_FDM * L_FDM

In the canonical LIBERO recipe, the best ratio is IDM:FDM = 0.125:1. The EMA target encoder is enabled with momentum 0.999, and Stage 1 runs for 3 epochs. Stage 2 then drops the EMA target, initializes from the Stage 1 checkpoint, and trains the flow-matching policy for 40 epochs.

LIBERO Data Recipe

The official SLIM recipe does not use the original LIBERO HDF5 files. It uses LeRobot v2.1 no-op-filtered datasets from the IPEC-COMMUNITY collection. There are five datasets:

Stage 1 uses libero_all_90, meaning the four target suites plus LIBERO-90. Stage 2 uses libero_all, meaning Object, Goal, Spatial, and LIBERO-10 without LIBERO-90. The datasets store two AV1 video streams at 256x256: observation.images.image and observation.images.wrist_image. Actions are 7D. The state is 8D, but the canonical config consumes the first seven state values.

Install the Training Environment

The reported training environment uses Python 3.12.9 and CUDA 12.4. You can inspect the repository and download checkpoints on an ordinary machine, but reproducing the paper recipe requires serious GPU capacity. The canonical run uses one node with eight H100 80GB GPUs. On smaller hardware, expect to adjust batch size, use gradient accumulation, or treat your run as an approximate reproduction.

git clone https://github.com/kzz1031/SLIM.git
cd SLIM
python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt
python -m pip install -e .

Then configure the main paths:

export LIBERO_DATA_ROOT=/path/to/lerobot/libero
export TORCH_HOME=/path/to/torch-cache
export DINOV2_MODEL_DIR=/path/to/dinov2-vitb14
export T5_MODEL_DIR=/path/to/t5-small
export SLIM_CACHE_DIR=/path/to/cache
export SLIM_LANGUAGE_CACHE=/path/to/language_embeddings

DINOv2 must be the official DINOv2-B/14 checkpoint loaded through the PyTorch Hub source expected by the repository. Do not substitute facebook/dinov2-base from Hugging Face, because its state dict does not match the architecture used by SLIM. T5-small should be downloaded as a complete snapshot, including tokenizer files, config files, and model.safetensors. Before downloading all datasets, run a local smoke test that loads DINOv2 and T5 from the configured paths.

Download Data and Cache Language

Download the five LIBERO datasets with the Hugging Face CLI:

python -m pip install huggingface_hub
mkdir -p "$LIBERO_DATA_ROOT"
for repo in \
  IPEC-COMMUNITY/libero_object_no_noops_1.0.0_lerobot \
  IPEC-COMMUNITY/libero_goal_no_noops_1.0.0_lerobot \
  IPEC-COMMUNITY/libero_spatial_no_noops_1.0.0_lerobot \
  IPEC-COMMUNITY/libero_10_no_noops_1.0.0_lerobot \
  IPEC-COMMUNITY/libero_90_no_noops_lerobot
do
  hf download "$repo" \
    --repo-type dataset \
    --local-dir "$LIBERO_DATA_ROOT/${repo#*/}"
done

Language can be encoded online, but caching T5 embeddings makes repeated training runs more convenient:

python scripts/precompute_language_embeddings.py \
  --data-root "$LIBERO_DATA_ROOT" \
  --data-mix libero_all \
  --output-dir "$SLIM_LANGUAGE_CACHE"

If video loading fails, check the torchvision_av backend first. The canonical recipe decodes the original videos directly. Extracted JPEG frames are useful for controlled ablations or debugging the input pipeline, but they are not the primary recipe.

Training Stage 1: Masked Trajectory Prediction

Stage 1 is what makes SLIM different from a direct flow-matching policy. The goal is not yet to train the final deployable controller. The goal is to learn latent interaction among current observations, future observations, and action chunks. The helper script assumes eight GPUs:

export WANDB_MODE=offline
bash scripts/train_stage1_8gpu.sh

Equivalent command:

torchrun --standalone --nproc-per-node=8 \
  -m slim.training.stage1 \
  --config configs/libero/stage1_idm0125_fdm1_h8.yaml

Important canonical settings:

If you only have a few GPUs, do not expect exact paper numbers. The useful learning path is still the same: verify data loading, check action and state shapes, train Stage 1 until a checkpoint is produced, then use that checkpoint to initialize Stage 2.

Training Stage 2: Flow-Matching Policy

Stage 2 turns the learned representation into a deployable policy. The model no longer receives the real future observation. It conditions on the current observation latent, proprioception, language embedding, and learned future-slot embeddings, then learns the velocity field used by flow matching.

bash scripts/train_stage2_8gpu.sh \
  checkpoints/stage1/<run>/checkpoints/epoch_3_pytorch_model.pt

Equivalent command:

torchrun --standalone --nproc-per-node=8 \
  -m slim.training.stage2 \
  --config configs/libero/stage2_policy_h8_40ep.yaml \
  --init-checkpoint checkpoints/stage1/<run>/checkpoints/epoch_3_pytorch_model.pt

The canonical Stage 2 run trains for 40 epochs, initializes from the Stage 1 checkpoint, keeps action horizon 8, uses global batch size 128, and uses 4 repeated flow-sampling steps at inference. If a distributed run is interrupted, prefer the repository's --resume-state or --auto-resume mechanism over loading only model weights. Optimizer state, scheduler state, RNG state, and data position all matter for reproduction.

Inference and LIBERO Evaluation

If you do not want to train from scratch, download the public checkpoint:

python -m pip install --upgrade huggingface_hub
export SLIM_RELEASE_ROOT="$PWD/checkpoints/releases"

hf download kzzwang/SLIM-LIBERO \
  --local-dir "$SLIM_RELEASE_ROOT/SLIM-LIBERO"

export SLIM_LIBERO_CHECKPOINT="$SLIM_RELEASE_ROOT/SLIM-LIBERO/checkpoints/epoch_40_pytorch_model.pt"

For evaluation, SLIM separates the policy environment from simulator environments. Standard LIBERO and LIBERO-Plus both install a Python package named libero, so they should not share a virtual environment.

After setting SLIM_TRAIN_PYTHON, SLIM_LIBERO_PYTHON, SLIM_PLUS_PYTHON, SLIM_LIBERO_HOME, SLIM_PLUS_HOME, and LIBERO_PLUS_CLASSIFICATION, run:

bash scripts/evaluate_all_8gpu.sh \
  "$SLIM_LIBERO_CHECKPOINT" \
  outputs/slim_libero_release \
  12000

The launcher runs standard LIBERO first and then LIBERO-Plus. It uses eight consecutive policy-server ports starting at 12000. The key outputs are:

outputs/slim_libero_release/
  libero/summary.txt
  libero_plus/summary.txt

For debugging a single suite, start a policy server:

python -m slim.serving.server \
  --checkpoint "$SLIM_LIBERO_CHECKPOINT" \
  --port 10093 \
  --bf16

Then run the LIBERO client:

TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD=1 \
python -m slim.evaluation.libero.evaluate \
  --checkpoint "$SLIM_LIBERO_CHECKPOINT" \
  --host 127.0.0.1 \
  --port 10093 \
  --task_suite_name libero_10 \
  --num_trials_per_task 50 \
  --action_chunk_size 8 \
  --send-state

Main Results

On original LIBERO, SLIM reports 97.5% overall success, which corresponds to 1,950/2,000 rollouts in the public reproducibility record. On LIBERO-Plus, the checkpoint trained on original LIBERO is evaluated zero-shot on 10,030 perturbation cases and reaches 77.45%, or 7,768/10,030. The LIBERO-Plus perturbations include camera changes, robot initial state shifts, language variation, lighting changes, background changes, sensor noise, and object-layout shifts.

A compact version of the paper's comparison:

Read this table carefully. SLIM does not win every column against every baseline; VLA-JEPA is slightly higher on LIBERO-Plus overall. The point is that SLIM is competitive while being much smaller, requiring no additional embodied policy or world pretraining, and targeting low-latency deployment. On CALVIN ABC to D, SLIM reaches 4.556/5 average sequence length, suggesting that the latent interaction policy also supports longer task composition beyond short LIBERO episodes.

The paper also reports real-world evaluation on five physical manipulation tasks under nominal, distractor, lighting, and background conditions. SLIM reaches average real-world progress 67.8, policy latency 77.3 ms, and policy-server GPU memory 2.01 GiB in the authors' measurement setup. These numbers support the practical claim: a smaller policy is not only easier to benchmark, but also easier to place inside an actual robot control stack.

Why Stage 1 Matters

The paper's ablations show that skipping Stage 1 and training the policy directly hurts robustness. The EMA target encoder is also important. In the reported ablation, enabling EMA improves LIBERO-Plus from 66.82% to 77.45% and CALVIN from 4.382 to 4.556. That makes sense because FDM predicts future latents; if the target encoder changes too quickly, the model learns against an unstable target.

The attention analysis is also useful. The authors project action-to-observation attention back to the image plane and compare SLIM with a policy trained without Stage 1. SLIM follows the manipulated object, gripper, and contact region more consistently throughout the manipulation sequence. This is qualitative evidence that the latent is not merely a generic visual embedding; it is shaped by action-relevant interaction.

Beginner Checklist

If this is your first SLIM run, avoid trying to reproduce everything on day one. A safer path is:

1. Clone the SLIM repository and create the Python 3.12.9 training environment.
2. Download the exact DINOv2-B/14 PyTorch Hub source and T5-small snapshot.
3. Run a local smoke test that loads DINOv2 and T5.
4. Download a small subset of LIBERO data first and verify schema, video decoding, action shape, and state shape.
5. Download kzzwang/SLIM-LIBERO before attempting full training.
6. Create separate Standard LIBERO and LIBERO-Plus environments.
7. Debug a small suite such as libero_10 with 50 trials per task.
8. Run evaluate_all_8gpu.sh only after the single-suite path works.
9. For training, run Stage 1 first, verify the checkpoint, then run Stage 2.
10. Record config hashes, seeds, batch size, GPU topology, and summary outputs.

The most common failure modes are mixed simulator environments, wrong DINOv2 weights, incomplete T5 snapshot, wrong LeRobot dataset version, or changed action and state formats. With LIBERO, a wrong BDDL path, asset path, or initialization-state path can break evaluation before the policy is even called.

When Should You Use SLIM?

SLIM is a good fit if you care about efficient manipulation policies, have demonstration data in a LeRobot-like format, want to evaluate on LIBERO, LIBERO-Plus, or CALVIN, or need a compact baseline against larger VLA models. It is also a strong case study for latent world modeling without test-time video generation: use future prediction as a training signal, but keep deployment simple.

SLIM does not answer every open question. The paper studies one compact model scale, so it does not fully establish how the method scales to many embodiments, larger robot datasets, or whole-body humanoid tasks. For industrial deployment, you still need safety, system-level latency management, sensor synchronization, recovery policies, and real-world domain adaptation. But for anyone learning VLA manipulation, SLIM is worth studying because it connects three important ideas in a clean way: self-supervised latent prediction, flow-matching action generation, and robustness evaluation on LIBERO.

Related Posts

• VLA Models overview
• LeRobot hands-on guide
• Run Embodied-R1.5-VLA on LIBERO