Run NEO-VLA on LIBERO: An Encoder-Free VLA Guide

Why a robot needs more than a scene description

Recognizing a cup is different from positioning a gripper around its rim. Manipulation depends on boundaries, local geometry, and small state changes that a semantic image representation may not preserve in a form an action policy can easily use. Most vision-language-action systems place a separately pretrained vision encoder before their language backbone. Action learning inherits that visual interface.

NEO-VLA, introduced in From Pixels to Actions—Towards Native Vision-Language-Action Policies by Weichen Fan and colleagues, explores another approach: let an action expert access visual computation at multiple backbone depths. The paper was submitted on November 4, 2026. This guide follows its November 9 v3 revision and the official VLAKit repository, checked on November 10, 2026.

Our practical objective is to serve the released checkpoint, connect LIBERO, complete a small rollout, and then run the default evaluation. The instructions below are grounded in upstream documentation and code. Reported scores belong to the authors; they are not measurements from a new reproduction performed for this article. For an introduction to the broader architecture, start with our OpenVLA explanation.

What encoder-free means here

Encoder-free does not mean that images magically become language tokens without computation. NEO-VLA uses a lightweight, two-stage convolutional patch embedder with an overall stride of 32. Each visual token corresponds to a 32 × 32 image region. The difference is the absence of a separate pretrained vision transformer before the shared backbone.

Visual and language tokens are processed together through the 40-layer NEO-2B backbone. Images retain their aspect ratios and are dynamically patchified within a budget of up to 88 visual tokens per view. Native Rotary Position Embedding represents time, height, and width separately. Text follows causal attention, while tokens within an image block can attend bidirectionally.

The prefix combines previous and current observations, up to three cameras at each time, camera intrinsics and rotations, a structured instruction, and robot proprioception. Unavailable arms or measurements are masked. A missing sensor reading is different from a measured zero, so replacing unknown values with zeros changes the input semantics.

Previous/current images ── patch convolution ──┐
Camera geometry + instruction + robot state ───┤
                                               v
                                   40-layer NEO-2B backbone
                                               |
                                Keys/values from selected layers
                                               v
                                   Flow-matching action expert
                                               |
                               Action chunk → simulator/controller

A separate, narrow action expert reads cached keys and values from selected backbone layers. Each expert layer also attends to its own action tokens. The default skip28 architecture connects 28 expert layers to layers distributed across the 40-layer backbone. It does considerably more than attach an MLP to the final language representation.

Continuous actions and the two meanings of horizon

During training, flow matching teaches the expert a velocity field between noisy actions and demonstration trajectories. At inference, a numerical solver follows that field to generate continuous action chunks. Actions do not have to be converted into words from a language vocabulary.

The general output layout supports 30 steps for two arms, with 26 dimensions per arm, plus an eight-dimensional body block. Arm channels include joint targets, end-effector translation and rotation increments, a gripper target, and hand joints. LIBERO uses only one arm and executes seven channels: three translation increments, three rotation increments, and the gripper.

The prediction horizon and executed horizon are different. A model can predict 30 rows while the LIBERO client executes only 16 before requesting another chunk. The released eval.json specifies a 20 Hz control rate, so 16 rows cover 0.8 seconds of simulated control. This does not mean the model is queried 20 times per second. Our guide to truncated flow matching explains why changing sampling steps is another, separate decision.

Source: NEO-VLA project page. The experiments identify useful action information early in the backbone, with different depth profiles for language, appearance, and geometry. They do not establish that removing all late layers is universally beneficial.

Step 1: prepare the policy environment

Use Linux, Conda, and a working NVIDIA CUDA setup. The package requires Python 3.10 or later, PyTorch 2.6 or later, and Transformers 4.50 or later. The authors measure inference on an RTX 3090 with 24 GB VRAM. Treat that as a reference configuration, rather than a guarantee that a smaller GPU will fit the complete stack.

Storage also matters. The released stage_libero/state.pt alone is approximately 12.6 GB, and the standalone backbone is an additional download. Begin in a new workspace. These commands download source archives without requiring Git operations:

mkdir -p "$HOME/neo-vla-lab"
export NEO_WORK="$HOME/neo-vla-lab"
cd "$NEO_WORK"
curl -fL https://github.com/EvolvingLMMs-Lab/NEO/archive/refs/heads/main.tar.gz \
  | tar -xz

conda create -n neo-policy python=3.10 -y
conda activate neo-policy
cd "$NEO_WORK/NEO-main/VLAKit"
python -m pip install --upgrade pip
# Example CUDA 12.4 wheels; use a compatible NVIDIA driver.
python -m pip install torch==2.6.0 torchvision==0.21.0 \
  --index-url https://download.pytorch.org/whl/cu124
python -m pip install -e . huggingface_hub
python -c 'import torch; print(torch.__version__, torch.cuda.is_available())'

If CUDA availability is false, fix the driver and PyTorch installation before loading the checkpoint. The optional .[prep] dependency group is for data preparation filters, not a prerequisite for this inference walkthrough. Archive URLs track moving branches; preserve the downloaded source and package versions when recording a reproducible experiment.

Step 2: download the matching artifacts

The official model card distinguishes native policy checkpoints from the standalone Hugging Face backbone. stage_libero is the simulation policy. stage2_backbone supplies the backbone configuration, tokenizer, modeling code, and weights used by the documented serving setup.

python - <<'PY'
import os
from huggingface_hub import snapshot_download

snapshot_download(
    repo_id="weepiess2383/NEO-VLA",
    allow_patterns=["stage_libero/*", "stage2_backbone/*"],
    local_dir=os.path.join(os.environ["NEO_WORK"], "checkpoints"),
)
PY
export NEO_VLA_BACKBONE="$NEO_WORK/checkpoints/stage2_backbone"
export NEO_LIBERO_CKPT="$NEO_WORK/checkpoints/stage_libero"
ls "$NEO_LIBERO_CKPT"

Keep state.pt, meta.json, embodiments.json, and eval.json together. The normalizer defines how actions and masks are interpreted for an embodiment. Substituting another checkpoint's normalizer can invalidate a rollout even when tensor shapes match. These native policy directories are not standard AutoModel.from_pretrained exports.

There is an important release limitation: the original NEOpp-2B-SFT initialization checkpoint is not public, according to the model card. stage2_backbone was extracted after joint training and is not an equivalent replacement for reproducing the original pretraining experiment. Use it for the documented evaluation path, and disclose any different initialization when conducting training research.

Step 3: start the inference service

Generate a server configuration from the checkpoint's evaluation specification. This avoids manually guessing camera slots, action frames, and embodiment settings:

cd "$NEO_WORK/NEO-main/VLAKit"
python -m neo_vla.bench.spec \
  --checkpoint "$NEO_LIBERO_CKPT" --device cuda \
  > "$NEO_WORK/server.json"
neo-vla-serve --config "$NEO_WORK/server.json" \
  --host 127.0.0.1 --port 8000

Keep this terminal running. From another terminal, check the HTTP endpoints:

curl -f http://127.0.0.1:8000/health
curl -f http://127.0.0.1:8000/metadata

A successful health response establishes that the service started. It does not establish task success. The service serializes predictions, so initially run one simulator client. This keeps rendering failures, model failures, and transport failures easier to diagnose.

Step 4: install LIBERO separately

The VLAKit benchmark guide specifies mujoco==3.2.3, robosuite==1.4.1, and numpy==2.2.6. LIBERO's older requirements file pins different versions. Installing that complete file would change the benchmark environment.

Create a second Conda environment for simulation. Export the workspace variables again because a new terminal does not inherit variables subsequently set in another terminal:

export NEO_WORK="$HOME/neo-vla-lab"
export NEO_LIBERO_CKPT="$NEO_WORK/checkpoints/stage_libero"
cd "$NEO_WORK"
curl -fL https://github.com/Lifelong-Robot-Learning/LIBERO/archive/refs/heads/master.tar.gz \
  | tar -xz
conda create -n neo-libero python=3.10 -y
conda activate neo-libero
python -m pip install --upgrade pip
python -m pip install numpy==2.2.6 mujoco==3.2.3 robosuite==1.4.1 \
  bddl==1.0.1 easydict pyyaml cloudpickle gym==0.25.2
python -m pip install --no-deps -e "$NEO_WORK/LIBERO-master"
# Install the client package without bringing policy training dependencies here.
python -m pip install --no-deps -e "$NEO_WORK/NEO-main/VLAKit"
export MUJOCO_GL=egl
export LIBERO_CONFIG_PATH="$NEO_WORK/libero-config"

This is a starting dependency set for evaluation, not a tested cross-platform lockfile. If an upstream import requires another dependency, follow its traceback and installation documentation while retaining the three benchmark pins. For systems without suitable EGL support, upstream permits MUJOCO_GL=osmesa; that requires the appropriate operating-system Mesa libraries.

LIBERO uses config.yaml to locate task definitions, initial states, and assets. Create it before the first import to avoid an interactive configuration prompt:

python - <<'PY'
import os
from pathlib import Path
import yaml

work = Path(os.environ["NEO_WORK"])
root = work / "LIBERO-master" / "libero" / "libero"
config = Path(os.environ["LIBERO_CONFIG_PATH"])
config.mkdir(parents=True, exist_ok=True)
paths = {
    "benchmark_root": str(root),
    "bddl_files": str(root / "bddl_files"),
    "init_states": str(root / "init_files"),
    "datasets": str(work / "LIBERO-master" / "libero" / "datasets"),
    "assets": str(root / "assets"),
}
(config / "config.yaml").write_text(yaml.safe_dump(paths))
from libero.libero.envs import OffScreenRenderEnv
print("LIBERO import OK")
PY

Evaluating an existing policy requires task definitions, initial states, and simulator assets. It does not require downloading the entire demonstration corpus for training. If a future source archive changes its layout, adjust these paths explicitly before running an experiment.

Step 5: run one episode, then the complete default protocol

Start with one task and one episode:

python -m neo_vla.bench.libero --benchmark libero \
  --checkpoint "$NEO_LIBERO_CKPT" \
  --server http://127.0.0.1:8000 \
  --libero-root "$NEO_WORK/LIBERO-master" \
  --suites libero_spatial --tasks 0 --episodes 1 \
  --out "$NEO_WORK/results/smoke"

A smoke test checks the complete path: rendering, observation construction, HTTP prediction, action execution, and result writing. One success does not validate the benchmark score, and one failure does not establish that the checkpoint is broken. Once the path works, use a new output directory for the full default run:

python -m neo_vla.bench.libero --benchmark libero \
  --checkpoint "$NEO_LIBERO_CKPT" \
  --server http://127.0.0.1:8000 \
  --libero-root "$NEO_WORK/LIBERO-master" \
  --out "$NEO_WORK/results/libero-full"

The released default covers four suites, each with ten tasks and 50 episodes per task: 2,000 episodes overall. Spatial, Object, Goal, and Long have step budgets of 220, 280, 300, and 520 respectively. Each episode first spends ten steps settling the scene; those steps do not consume its task budget.

Results include settings.json, per-task JSON files, and summary.json. The client skips tasks whose result files already exist. This supports resuming an interrupted experiment, but it also means that reusing a directory after changing the checkpoint or protocol can mix incompatible results. Keep smoke tests, full evaluations, and ablations in separate directories.

Why the simulator adapter matters

The LIBERO checkpoint predicts actions in the wrist camera's frame. The client reads the live camera pose, transforms increments into the robot base frame, and applies suite-specific unit factors. Scene and wrist images are rendered at 256 × 256 and oriented according to the evaluation specification. The model's gripper convention also needs conversion before execution in robosuite.

Skipping these conversions can produce a policy that loads successfully but moves in the wrong direction. Begin with the official client before writing a custom adapter. Understand centimetres, radians, camera order, and action-column selection first. For broader evaluation context, our LIBERO-PRO planning guide discusses why distribution shifts deserve separate attention.

Training recipes and reproduction boundaries

Source: the VLAKit training guide. The described corpus contains 2,231,583 episodes and approximately 928.4 million rows across nine dataset families. This scale explains why checkpoint inference is a better first exercise than pretraining.

Stages 1 and 2 use a reference distributed shape of 16 nodes with eight GPUs each. Setting the GPU count to one does not turn the recipe into an inexpensive laptop experiment: the large global batch remains, and gradient accumulation must divide correctly across the world size.

Inspect launcher arguments before starting training:

# In neo-policy, from NEO-main/VLAKit.
# Replace these paths with an actually prepared corpus and output location.
export NEO_VLA_DATA_ROOT=/data/robot-corpus
export NEO_VLA_OUTPUT_ROOT=/data/neo-runs
bash scripts/stage1.sh --print-argv

This only prints the command. It does not supply a missing initialization checkpoint or build a training corpus. Upstream says the pretraining dataset upload is incremental and incomplete. Source-specific converters and complete manifest, prompt-cache, and normalizer builders are not included. The published guide does not document a one-command LIBERO demonstration-to-training pipeline.

For adaptation research, the model card supports initializing from stage2 using trainer.init_from. Download that additional artifact and prepare a schema-3 manifest, the expected embodiment normalizer, and compatible processed data. Tokenizer identifiers, prompt templates, frame references, and control frequency must agree. The supplied posttrain_real.sh launcher targets real-robot adaptation; it is not an automatic LIBERO training script.

Stages 2 and 3 also use multimodal understanding co-training. The example code uses COCO ShareGPT with one unpacked chat row per robot row and a chat loss weight of 0.1. The model card explicitly distinguishes this example from the original understanding mixture. Replacing the original mixture changes the experiment. Removing understanding data altogether changes it even more: the paper reports substantially worse RoboTwin Clean2Rand performance after action-only post-training.

Read the results with their evaluation settings

Tables 2 and 7 of paper v3 describe different experiments:

The controlled backbone comparison uses final 30,000-step checkpoints and 500 episodes per suite. The main benchmark reports a 27,500-step checkpoint and 2,000 episodes per suite. The released runner defaults to 500 episodes per suite. Therefore, a default evaluation is not automatically an exact reproduction of the 97.4% paper result. Matching episode counts alone is insufficient; checkpoint identity, initial-state sampling, simulator versions, camera settings, and horizons also matter.

The paper reports 89.7% overall on LIBERO-plus but 47.9% average in its LIBERO-PRO table. Position and task-goal perturbations remain difficult. Strong perceptual and language robustness does not imply arbitrary spatial or task generalization. Likewise, the backbone comparisons do not isolate encoder removal as the sole causal factor because pretraining and model capacity also differ.

Source: NEO-VLA project. Each task uses 50 training demonstrations and 20 evaluation trials. NEO-VLA achieves 12/20, 17/20, and 13/20 respectively. These small experiments demonstrate research capability rather than industrial deployment reliability.

Troubleshooting and a useful completion target

When simulator imports fail, check the active Conda environment and pinned versions first. When rendering fails, diagnose EGL or Mesa before changing the model. HTTP connection failures point to service startup or addressing; HTTP 500 responses require the policy-side traceback.

For actions that look geometrically wrong, inspect image orientation, camera slots, coordinate frames, and gripper conversion before increasing solver steps. If rerunning seems suspiciously fast, check whether task result files caused the resume logic to skip work. A health response, one successful episode, and a completed benchmark represent three different levels of evidence.

A useful beginner milestone is a running service, a recorded smoke test, a full evaluation with saved settings, and an explanation of how its protocol differs from the paper. After that, change one variable at a time, such as the executed horizon, while retaining a separate result directory. That turns a model download into an interpretable robotics experiment.

Related Posts

• OpenVLA: understand Vision-Language-Action foundations
• SmolVLA: practical policy training on a personal GPU
• Flow matching and truncated sampling for VLA