NEO-VLA: Encoder-Free VLA Achieves 95.3% on LIBERO

Read alongside the LIBERO setup walkthrough.

Every VLA (Vision-Language-Action) model built in the last two years has shared one silent assumption: you need a strong, separately pretrained vision encoder — ViT, SigLIP, DINOv2 — to "understand" images before the language backbone takes over. NEO-VLA flips that assumption. The paper "From Pixels to Actions: Towards Native Vision-Language-Action Policies" from S-Lab (Nanyang Technological University), SenseTime Research, and Ropedia shows that removing the separate encoder entirely — letting one backbone learn both vision and language from scratch — not only preserves performance but achieves 95.3% on LIBERO with full fine-tuning and 90.9% with a completely frozen backbone (only the action expert trained).

This is a structural rethinking of VLA design, not a minor engineering trick.

Why Remove the Vision Encoder?

The standard VLA pipeline looks like this:

Image → [ViT/SigLIP Encoder] → visual tokens
Text  → [tokenizer]           → text tokens
[visual + text tokens] → LLM backbone → action prediction

This design has three structural weaknesses:

1. Frozen visual representations — SigLIP was pretrained for image-text matching, not wrist-level motor control. When the robot needs to distinguish a 2° wrist rotation, general-purpose features may not carry that signal.
2. Depth mismatch — the backbone has 40 layers, but visual features enter only at layer 0. All the visual information must propagate through the entire LLM stack, which was never designed to refine visuomotor features.
3. Double model overhead — loading two separate model weights at inference time.

NEO-VLA asks: what if we build one backbone that learns vision and language jointly from the start?

Architecture

The architecture has three components:

1. Lightweight Patch Embedder

Instead of a ViT-Large with 300M parameters, NEO-VLA uses a few lightweight convolutional layers to convert image patches into tokens. These layers train jointly with the backbone — no separate pretraining step.

# Conceptual sketch — the embedder is intentionally minimal
class PatchEmbedder(nn.Module):
    def __init__(self, patch_size=16, hidden_dim=2048):
        super().__init__()
        self.proj = nn.Conv2d(3, hidden_dim, kernel_size=patch_size, stride=patch_size)
        self.norm = nn.LayerNorm(hidden_dim)

    def forward(self, x):  # x: (B, 3, H, W)
        tokens = self.proj(x)           # (B, D, H/p, W/p)
        tokens = tokens.flatten(2).T    # (N_patches, B, D)
        return self.norm(tokens)

Critically, patch tokens and word tokens enter the backbone together at layer 0 — there is no separate visual preprocessing.

2. Shared Vision-Language Backbone (2B parameters)

A 40-layer decoder-only transformer processes everything as one unified prefix:

• Current and previous multi-view images (patch tokens)
• Language task description (word tokens)
• Robot state (joint angles, end-effector pose)

No separate visual pathway. One sequence, one attention stack.

3. Multi-Layer Flow-Matching Action Expert

This is the key innovation. Instead of reading only the final layer hidden state, the action expert reads cached keys and values from multiple backbone layers.

Layer  1 → K, V cache ──────────────────────┐
Layer 10 → K, V cache ──────────────────────┤
Layer 20 → K, V cache ──────────────── Action Expert → action chunk
Layer 30 → K, V cache ──────────────────────┤
Layer 40 → K, V cache ──────────────────────┘

The motivation comes from layer-wise probing experiments: the paper shows that action-critical visual signals often peak at intermediate layers, not the final one. The last layer is biased toward semantic/language reasoning — not fine-grained visuomotor control.

The action expert uses flow matching (not diffusion) to predict the action trajectory, requiring only 1-5 NFE (Number of Function Evaluations) compared to 20-100 denoising steps for diffusion-based policies.

Results on LIBERO

LIBERO is the standard tabletop manipulation benchmark with four suites:

• LIBERO-Spatial — spatial understanding (left/right, above/below)
• LIBERO-Object — recognizing diverse objects
• LIBERO-Goal — reaching specified goals
• LIBERO-Long — long-horizon action sequences

Three numbers worth highlighting:

1. 90.9% frozen is the most striking result — the backbone is never updated, only the action expert trains. The visual representation NEO-VLA develops from scratch is already good enough for manipulation without domain adaptation.
2. NEO-VLA (2B params) outperforms Qwen3-VL at the same parameter count, in both settings.
3. The 2.8% gap between 95.3% and 92.5% sounds small, but above 90% every percent is hard-won.

On RoboTwin 2.0 (simulation benchmark with randomized object variations):

Demo Tasks

The paper demonstrates NEO-VLA across a range of manipulation tasks:

• Pick and place with randomly positioned objects
• Stacking with language-specified ordering ("put the red block on the blue block")
• Cabinet open and retrieve — requires 3D spatial reasoning
• Bimanual coordination — two-arm tasks requiring synchronized movement

In all tasks the model receives only raw RGB images plus a language instruction — no depth, no point cloud, no 3D reconstruction.

Installation and Inference

NEO-VLA is released in the official EvolvingLMMs-Lab repository: github.com/EvolvingLMMs-Lab/NEO.

Environment Requirements

# Prerequisites
Python >= 3.10
CUDA >= 12.1
GPU: RTX 3090 24GB or better (inference)
     A100 80GB for full fine-tuning

# Clone and install
git clone https://github.com/EvolvingLMMs-Lab/NEO
cd NEO
pip install -e ".[vla]"
# or manually:
pip install torch torchvision transformers accelerate
pip install lerobot   # for LIBERO data loading

Download Weights

# NEO-VLA 2B model via Hugging Face CLI
huggingface-cli download EvolvingLMMs-Lab/NEO-VLA-2B \
  --local-dir ./checkpoints/neo-vla-2b

# or in Python
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id="EvolvingLMMs-Lab/NEO-VLA-2B",
    local_dir="./checkpoints/neo-vla-2b"
)

Basic Inference

import torch
from neo_vla import NEOVLAPolicy

# Load policy
policy = NEOVLAPolicy.from_pretrained(
    "./checkpoints/neo-vla-2b",
    action_readout_layers=[10, 20, 30, 40],  # which layers the action expert reads
    device="cuda"
)
policy.eval()

# Prepare observation
obs = {
    "image": {
        "top":   torch.rand(1, 3, 224, 224).cuda(),   # top-down camera
        "wrist": torch.rand(1, 3, 224, 224).cuda(),   # wrist camera
    },
    "language_instruction": ["put the red block on the plate"],
    "robot_state": torch.zeros(1, 7).cuda(),   # joint angles
}

# Predict action chunk
with torch.no_grad():
    action = policy.predict_action(obs)
    # shape: (1, action_horizon, action_dim)
    # action_horizon = 50 steps, action_dim = 7 (6DOF + gripper)
print(f"Action chunk: {action.shape}")  # torch.Size([1, 50, 7])

Evaluating on LIBERO

# Run standard LIBERO evaluation
results = policy.evaluate_libero(
    suite="libero_goal",
    num_episodes=50,
    render=False,   # set True to watch the robot
)
print(f"Success rate: {results['success_rate']:.1%}")
# Expected output: ~90.9% (frozen backbone)

Fine-Tuning on Your Own Data

NEO-VLA supports action-expert-only training, which keeps the backbone frozen and only optimizes the lightweight action expert. This dramatically lowers the VRAM requirement.

Preparing Your Dataset

Format data to the LeRobot standard:

from lerobot.common.datasets.push_dataset_to_hub import push_dataset_to_hub

# Your dataset should contain:
# - observation.images.{camera_name}: video frames
# - observation.state: robot joint states
# - action: joint velocities or positions
# Each episode is typically 100-500 steps

push_dataset_to_hub(
    raw_dir="./my_teleoperation_data/",
    repo_id="my-org/my-task",
    raw_format="rlds",   # or "pusht", "xarm", etc.
)

Action-Expert-Only Training (Lowest VRAM)

# VRAM: ~20 GB for the 2B model
python train_neo_vla.py \
  --model_path ./checkpoints/neo-vla-2b \
  --dataset lerobot/libero_goal \
  --train_mode action_expert_only \
  --readout_layers 10,20,30,40 \
  --batch_size 32 \
  --num_steps 50000 \
  --lr 1e-4 \
  --output_dir ./outputs/my-task-expert

Full Fine-Tuning (Highest Accuracy)

# VRAM: ~80 GB — requires an A100 80 GB or multi-GPU setup
python train_neo_vla.py \
  --model_path ./checkpoints/neo-vla-2b \
  --dataset lerobot/libero_goal \
  --train_mode full_finetune \
  --readout_layers 10,20,30,40 \
  --batch_size 16 \
  --num_steps 100000 \
  --lr 5e-5 \
  --gradient_checkpointing \
  --output_dir ./outputs/my-task-full

Practical tip: For new tasks in a domain visually similar to LIBERO, action-expert-only training is usually sufficient and runs much faster. Full fine-tuning is worth the cost only when the visual domain is substantially different (outdoor environments, unusual lighting, etc.).

Why NEO-VLA Matters Beyond the Numbers

Encoder-free ≠ visually blind

Dropping the vision encoder doesn't mean the model stops understanding images. When a 2B backbone trains end-to-end with both visual and language data, it learns visual representations optimized for robot control — not for general-purpose image-text matching. The objective is tighter, so the representations are better-suited to the task.

Multi-layer readout challenges a core assumption

The finding that action-critical features peak at intermediate layers, not the final one, has implications for every VLA currently in production. If you are using Pi0, OpenVLA, or any Qwen-VL-based policy and reading only the last hidden state, you may be discarding significant visuomotor information.

Stronger robustness under visual shift

The paper reports that NEO-VLA maintains higher success rates under visual perturbations — changed background colors, different lighting, shifted camera angle — than encoder-based models. The co-trained backbone learns features that are more invariant to these surface-level changes precisely because it was never optimized for general-purpose visual recognition.

Comparison with Other VLA Architectures

Three architectural pillars where NEO-VLA differentiates:

• No separate encoder → simpler pipeline, fewer parameters for equivalent performance
• Multi-layer readout → richer visuomotor signal extraction from the backbone
• Flow matching → faster inference than diffusion (1-5 NFE vs. 20-100 NFE)

Known Limitations

1. Tabletop-only benchmarks — no public real-robot deployment videos for more complex scenarios (mobile manipulation, outdoor environments).
2. Pretraining dataset not fully disclosed — the paper mentions an "action pretraining dataset" but does not detail its scale or sources.
3. Embedded deployment not yet benchmarked — inference on Jetson Orin remains uncharted; Pi0 Fast already has numbers for embedded deployment, NEO-VLA does not.
4. Static frames only — the current implementation uses per-frame images, not temporal video sequences as input, unlike some newer VLA models.

Summary

NEO-VLA asks the right question: why maintain two separate encoders when a single unified backbone can learn better representations? The 95.3% LIBERO result — and especially the 90.9% with a fully frozen backbone — demonstrates that native VLA architecture is not just a theoretical appeal. It works.

The multi-layer flow-matching action expert also opens a new research direction: where inside a backbone should VLA models extract visual information? The answer, it turns out, is not always the last layer.

Paper and code: arxiv.org/abs/2611.06918 | github.com/EvolvingLMMs-Lab/NEO

Related Posts

• GroundingPI: XPENG's 4B VLA Grounding Model Beats GPT-6 Astra
• A1-VLA: Truncated Flow Matching for Robot Manipulation
• AgenticRobo RL: Planner VLA on LIBERO-Pro