NS-VLA v2 is worth studying if you fine-tune Vision-Language-Action models for manipulation and keep seeing the same failure pattern: the model knows the scene, but it loses the order of operations. Instead of asking a VLA to directly emit continuous actions from an image and an instruction, NS-VLA inserts neuro-symbolic primitives between reasoning and control. The policy predicts structured steps such as move, pick, place, push, open, close, and release, then a low-level primitive solver turns that symbolic step into an action chunk.
The original paper is NS-VLA: Neuro-Symbolic Vision-Language-Action Model for Embodied Reasoning and Manipulation — arXiv v2, 2026. The implementation is available in Zuzuzzy/NS-VLA, with model weights under Zuzuzzy/NS-VLA and data under Zuzuzzy/NS-VLA-Dataset on Hugging Face. This guide walks through the idea, architecture, installation, BC warmup, GRPO/AWR fine-tuning on LIBERO, inference, and the reported results.
If you are new to VLA policies, read OpenVLA: an open VLA for robots first. If you already understand RL fine-tuning, TGRPO: fine-tuning VLA with Trajectory GRPO gives a useful comparison point for the GRPO part of NS-VLA v2.
What Problem Does NS-VLA Solve?
VLA manipulation commonly fails for three reasons. The first is long-horizon reasoning: an instruction such as "put the right bowl into the tray and close the drawer" contains several stages, but a raw action vector at one timestep does not expose which stage the policy believes it is executing. The second is object binding: when several objects look similar, the model may confuse "left bowl" and "right bowl." The third is RL credit assignment: an episode-level success reward tells you that the rollout failed, but not whether the bad decision was the primitive, the object argument, the gripper action, or the final placement.
NS-VLA changes the interface from:
image + language + proprioception -> continuous action
to:
image + language + state -> symbolic primitive + argument
symbolic primitive + argument + state -> action chunk
The primitive layer is not a hand-written classical planner. The primitive is still selected by a neural policy from visual and language context. The difference is that the high-level output is structured. That makes it easier to supervise, inspect, reward, and debug. For a beginner, a primitive is best understood as a named macro-action: pick(red_mug), place(red_mug, tray), or open(drawer). The low-level solver then handles continuous control.
The project pipeline shows the two-layer design clearly. The encoder processes observation and instruction. The monotone pointer tracks progress through the primitive sequence. The primitive solver receives the current primitive, target object, and robot state, then predicts the continuous action chunk. This is why the method is called neuro-symbolic: perception and selection are learned, while primitives and arguments provide symbolic structure.
Architecture: Encoder, Pointer, Solver
There are three parts to keep in mind when reading the code.
1. Multi-modal encoder. The encoder consumes RGB observations, language instruction, and proprioception. Depending on the configuration, the visual-language backbone can reuse a pretrained VLA or a compatible VLM encoder. The encoder must represent both the scene and the task progress. If your setup has both static and wrist cameras, keep camera ordering identical across data preparation, training, and inference. Swapping camera order is a subtle bug because the model still runs, but attention learns the wrong viewpoint.
2. Symbolic primitive pointer. NS-VLA uses a monotone pointer to map episode progress onto primitive steps. "Monotone" matters because manipulation tasks usually move forward through a sequence. After a successful place, the policy should not freely jump back to pick unless the task actually requires recovery. The pointer acts as a progress tracker and reduces the search space.
3. Low-level primitive solver. The solver takes the selected primitive and emits an action chunk. In LIBERO manipulation, the action is typically an end-effector delta pose plus a gripper command. Chunking improves stability because the model does not need to run a full VLA forward pass for every tiny movement. It can predict, for example, an 8-step chunk, execute it, observe again, then update the primitive if needed.
A simplified trace for "put the red mug in the cabinet" might look like this:
Instruction: "Put the red mug in the cabinet"
Step 0-18: move(red_mug)
Step 19-35: pick(red_mug)
Step 36-70: move(cabinet)
Step 71-92: place(red_mug, cabinet)
Step 93-110: release(red_mug)
Pure behavior cloning only sees state-action pairs. NS-VLA also sees the primitive trace, so it knows that steps 19-35 are a pick phase rather than just another local end-effector motion.
Stage I: BC Warmup
Training starts with behavior cloning. Do not skip it. GRPO/AWR only works well when the initial policy can already produce meaningful rollouts. If the policy is random, every rollout fails, advantages are noisy, and RL updates push the model away from useful demonstrations.
The repository workflow starts by preparing 1-shot data from LIBERO:
git clone https://github.com/Zuzuzzy/NS-VLA.git
cd NS-VLA
conda create -n nsvla python=3.10 -y
conda activate nsvla
pip install -e .
pip install -r requirements.txt
python data/prepare_1shot.py \
--suite libero_spatial \
--output_dir data/libero_1shot
You also need LIBERO and MuJoCo installed as described by the repo and paper. Before starting any training job, check the environment:
python -c "import mujoco, libero; print('env ok')"
python -c "import torch; print(torch.cuda.is_available())"
For beginners, the most common problem is not the model; it is the dataset path. Use explicit environment variables:
export LIBERO_DATA_DIR=/path/to/LIBERO/datasets
export NSVLA_DATA_DIR=$PWD/data/libero_1shot
export WANDB_MODE=offline
Then run BC warmup with the repository training script:
bash scripts/train.sh \
--stage bc \
--suite libero_spatial \
--data_dir $NSVLA_DATA_DIR \
--output_dir runs/ns-vla-bc-spatial \
--batch_size 8 \
--epochs 10
Flag names may change across commits, so read scripts/train.sh and the matching config before launching a long run. The technical requirement is stable learning for both primitive classification or pointer loss and action regression loss. If primitive accuracy remains low, later RL cannot fix much because the model does not know which stage of the task it is executing.
Stage II: GRPO for High-Level, AWR for Low-Level
The interesting part of NS-VLA v2 is the split RL objective. High-level primitive decisions are optimized with GRPO because they are discrete structured choices that can be compared across rollout groups. The low-level continuous solver is optimized with AWR because it needs a softer regression-style update that does not destroy the BC checkpoint.
GRPO stands for Group Relative Policy Optimization. It estimates advantage by comparing samples inside a group rather than training a separate critic. In robotics, this is attractive when reward is noisy but you can run multiple rollouts for the same task. A configuration such as group_size=8 means: for a task distribution, sample eight rollouts, score them, and update the policy toward the better relative outcomes.
AWR stands for Advantage Weighted Regression. It weights imitation-style action regression by advantage. If an action chunk leads to useful progress, clone it more strongly. If it causes the robot to miss a grasp or place the object incorrectly, reduce its weight. Because AWR remains close to regression on policy-generated data, it is usually less brittle than direct policy gradient on continuous robot actions.
The Stage II loop can be read as:
1. Load the BC checkpoint.
2. For each LIBERO task, generate a group of 8 rollouts.
3. Compute rewards from success, subgoal progress, and primitive completion.
4. Use GRPO to update the primitive pointer and high-level decisions.
5. Use AWR to update the continuous primitive solver.
6. Keep a KL anchor so the model does not drift too far from BC.
7. Evaluate periodically on LIBERO-Spatial/Object/Goal/Long.
A typical command looks like this:
bash scripts/train.sh \
--stage rl \
--suite libero_spatial \
--ckpt runs/ns-vla-bc-spatial/checkpoint-last \
--output_dir runs/ns-vla-rl-spatial \
--group_size 8 \
--chunk_H 8 \
--algo grpo_awr \
--kl_coef 0.02
If you only have one consumer GPU, reduce batch size and parallel environments first. Do not reduce group_size too aggressively because GRPO needs within-group comparison. If memory is still the bottleneck, use gradient accumulation, LoRA or QLoRA, and train one LIBERO suite at a time. For a serious reproduction, run three jobs: a 100-episode smoke test, a paper-style run, and a task-specific ablation.
Reward and Logging
Do not track only final success rate. For a neuro-symbolic policy, log at least six groups of metrics:
| Metric group | What it tells you |
|---|---|
success_rate |
Task completion rate on LIBERO |
primitive_accuracy |
Whether the pointer selects the expected primitive |
primitive_switch_count |
How often the policy changes primitives in one episode |
subgoal_completion |
Whether each primitive reaches its local goal |
action_l2 |
Whether the solver still stays near demonstration actions |
kl_to_bc |
Whether RL drifts too far from the BC checkpoint |
If success rate rises but primitive_switch_count becomes unstable, the policy may be spamming primitive changes. If kl_to_bc spikes early, reduce learning rate or increase the KL coefficient. If primitive accuracy is good but action error is high, focus on the solver, action representation, gripper threshold, and proprioception normalization rather than the high-level pointer.
Always save rollout videos. In LIBERO you will usually see four failure modes: wrong object, contact without grasp, correct grasp with wrong placement, or correct primitive order but timeout. These require different fixes. Better visual data helps wrong-object failures. Gripper threshold tuning helps grasp failures. Longer horizon, better subgoal reward, or curriculum helps timeout failures.
Inference: From Instruction to Action
Once the RL checkpoint is ready, inference is conceptually simple, but preprocessing must match training exactly:
from nsvla.policy import NSVLAPolicy
from libero.envs import make_libero_env
policy = NSVLAPolicy.from_pretrained(
"runs/ns-vla-rl-spatial/checkpoint-best",
device="cuda",
)
env = make_libero_env("libero_spatial", task_id=0)
obs = env.reset()
instruction = env.get_language_instruction()
done = False
while not done:
action_chunk, info = policy.predict_action_chunk(
obs=obs,
instruction=instruction,
horizon=8,
)
for action in action_chunk:
obs, reward, done, env_info = env.step(action)
if done:
break
Treat this as a skeleton for understanding the flow, not a guaranteed API for every repository commit. In practice, start from the official eval script, confirm that the checkpoint works, then wrap it into your own service. Preserve image normalization, proprioception normalization, action scaling, camera order, and language templates. A mismatch in any of those can erase the gains from fine-tuning.
For real robots, add a safety layer below the policy: velocity limits, workspace bounds, a lightweight collision check, force/torque thresholds, and an emergency stop path. NS-VLA is easier to debug than a fully opaque end-to-end policy, but it is still a learned policy. Do not blindly execute an 8-step action chunk if sensors show that the robot has already collided or lost the object.
Results on LIBERO and Real Robots
According to the paper and project page, NS-VLA v2 reports strong LIBERO performance and real-world manipulation demos. The project page highlights 98.6% on the full LIBERO benchmark and 79.4% on LIBERO-Plus, while the arXiv v2 paper includes additional 1-shot, ablation, and real-world results. The important interpretation is that NS-VLA is not merely scaling the backbone. Its core contribution is the primitive structure that makes long-horizon reasoning easier to learn and easier to inspect.
When you evaluate your own run, read each LIBERO suite separately. LIBERO-Long exposes ordering and memory problems. LIBERO-Object exposes object grounding mistakes. LIBERO-Spatial exposes spatial relation errors. If your model performs well on Object but poorly on Long, adding more primitive trace supervision and long-task curriculum is likely more useful than simply increasing BC epochs.
Beginner Reproduction Checklist
- Clone the repository and create the correct Python/CUDA environment.
- Install LIBERO, MuJoCo, PyTorch, and repository dependencies.
- Download dataset and checkpoint assets from Hugging Face.
- Run the provided eval script before training anything.
- Prepare 1-shot or full demonstration data.
- Train BC warmup until primitive and action losses stabilize.
- Run GRPO/AWR RL with grouped rollouts, videos, and KL logging.
- Evaluate each LIBERO suite separately.
- Classify failures from videos, not only from scalar metrics.
- Move to real robots only after simulation behavior is stable.
If you have tried VLA-Adapter: train a 0.5B VLA with 9.6GB VRAM, the difference is clear. VLA-Adapter asks how small a capable VLA can be. NS-VLA asks how much structure we should add so that a VLA can reason through multi-step manipulation and remain debuggable.
When Should You Use NS-VLA?
Use NS-VLA when your task has multiple steps, multiple objects, and order-sensitive manipulation. Examples include opening a drawer, taking an object, placing it into a container, and closing the drawer. These tasks fit primitives because failures usually belong to a clear stage. NS-VLA also helps teams that need understandable logs: "the policy selected the wrong place primitive" is much easier to act on than "action dimension 4 drifted."
Do not start with NS-VLA if your task is a single-step pick-and-place, your dataset is tiny, and your evaluation pipeline is not stable yet. A simpler behavior cloning or OpenVLA fine-tuning baseline may be enough. The neuro-symbolic layer adds complexity: primitive vocabulary, trace alignment, reward design, grouped rollout infrastructure, and two-level optimization. That complexity is justified when the task is long-horizon or when debugging matters.
Conclusion
NS-VLA v2 is a strong example of where robot learning is moving: large end-to-end VLA models are useful, but structure still matters. The paper brings symbolic primitives back without reverting to a brittle hand-coded planner. BC warmup gives the model a reasonable starting policy. GRPO improves high-level primitive selection. AWR improves the low-level continuous solver while keeping updates close to useful behavior.
For beginners, the right way to learn NS-VLA is to reproduce it on LIBERO first. Run eval, watch videos, inspect primitive traces, then adjust rewards or horizons. Once you can look at a failed episode and say "wrong primitive," "wrong object argument," or "solver approached too low," you have reached the real value of NS-VLA: turning VLA manipulation from a black box into a pipeline you can reason about.
