A robot may recognize the correct object yet produce a poor motion as its gripper approaches it. For a Vision-Language-Action model, recognizing an object early in the network does not guarantee that the object still strongly influences the final action representation. STAR, short for STeering Action Representations, investigates this gap and modifies inference while keeping policy weights frozen.
This tutorial covers the publicly released implementation in the official STAR repository: OpenPI π0.5 on LIBERO-PRO. We will explain the mechanism, separate the policy server from the simulator environment, run native inference, enable STAR and plan a fair evaluation. No additional policy training is required to try STAR with the supplied compatible checkpoint.
Sources were checked on November 3, 2026, including the repository, project website and arXiv v1 paper. Their π0.5 results and hyperparameters differ, so we identify those versions explicitly below. Reported benchmark numbers belong to the authors; we have not rerun their GPU experiments.
1. What you need to understand first
A VLA consumes camera images, a language instruction and robot state, then generates actions. In π0.5, a vision-language backbone builds a multimodal representation, while an action expert uses flow matching to generate an action chunk: several predicted future actions. The controller executes part of that chunk, obtains a new observation and replans.
Proprioception describes the robot itself, including the end-effector position, orientation and gripper state. Images describe what the camera sees; proprioception describes the robot's configuration and recent movement. STAR uses this second source of information to decide how strongly to edit late representations.
LIBERO-PRO tests generalization under changes to language, object attributes, spatial placement, task goals and visual context. It should not be confused with the original LIBERO benchmark. A high success rate on familiar scenes does not establish robustness when objects move or instructions change.
For a first attempt, aim to finish one episode in native mode and one with STAR. Full evaluation comes later. You need Linux, an NVIDIA GPU, compatible drivers and space for checkpoint downloads. The README cites an upstream inference estimate of more than 8 GB GPU memory and reports testing on Ubuntu 22.04. Treat that as an estimate rather than a promise that any particular 8 GB card will work.
Budget for inference memory, model downloads and simulator rollouts before considering fine-tuning hardware. Testing a method advertised as training-free should begin with the existing checkpoint and a small evaluation, not a new training run.
2. The paper's idea: where does object evidence weaken?
The authors probe the model with paired observations. One image is unchanged; in the other, pixels belonging to the task-relevant object are replaced with the image's mean RGB value. They compare action hidden states at each transformer layer between the two inputs.
If an action representation depends on the object, removing its visual evidence should change that representation. The paper measures a normalized L2 difference as an object-awareness score. For the tested π0.5 model, this sensitivity rises through early and middle layers, then falls near the output. This is empirical evidence of a late-layer grounding gap, rather than a theorem about every VLA architecture.
STAR carries a trend in residual updates across late layers. Think of each layer as proposing an adjustment to the current representation. Instead of accepting only the current adjustment, STAR combines it with a running trend from preceding layers in the intervention region. The intended effect is to preserve useful object-aware directions until action decoding.
Object masking is a diagnostic used in the research. Ordinary STAR inference does not require a segmentation mask, an object detector or manually annotated target regions. That distinction matters when estimating the work needed to run the released code.
3. Architecture: depth-wise EMA with proprioceptive scheduling
A conventional residual block adds an update to its input hidden state. STAR replaces the late-layer update with an exponential moving average, or EMA:
delta_k = output_k - input_k
momentum_k = beta_t * momentum_previous_layer
+ (1 - beta_t) * delta_k
corrected_output_k = input_k + momentum_k
This EMA runs across layers within a forward pass. It is not a temporal low-pass filter applied to motor commands. The residual momentum and the history of proprioceptive observations serve different purposes. Averaging consecutive output actions would therefore implement a different algorithm.
The general paper design leaves the first two-thirds of transformer layers unchanged and mixes updates in the final third. The released π0.5 recipe specifically targets the last six action-expert layers. Its controller initializes momentum using the first intervention layer's own delta. Consequently, the first corrected update matches the original update rather than shrinking it by mixing against a zero vector. See controller_core_v2.py for the implementation.
The coefficient beta_t controls how much accumulated momentum survives. A larger value keeps more of the preceding update trend. A smaller value gives the current layer more influence, making the intervention more conservative during reorientation or precise adjustments.
Four signals drive the scheduler
| Signal | Meaning | Contribution in the README recipe |
|---|---|---|
c_t |
Cosine similarity of consecutive displacements | Consistent movement supports stronger mixing |
v_t |
End-effector displacement magnitude | Translation contributes positively |
j_t |
Difference between consecutive displacements | Abrupt change contributes negatively |
w_t |
Orientation change | Rotation contributes negatively |
Although the code uses the name j_t, its expression is the norm of a difference between displacements. The paper describes an acceleration-like change. Avoid interpreting it as a physically calibrated third time derivative with standard jerk units.
The scheduler normalizes individual signals against past observations, combines them using signed weights, then normalizes the resulting score using a median and interquartile range. A sigmoid maps that score into a bounded beta interval. The released scheduler.py also clips individual normalized signals and handles insufficient history. It uses no future trajectory or oracle phase labels.
4. Install the π0.5 policy server
The following commands are instructions to run on your own machine. Install uv using its official documentation and confirm that nvidia-smi sees your GPU. Resolve driver problems before downloading model dependencies.
git clone --recurse-submodules https://github.com/PKU-VaLuE-Lab/STAR.git
cd STAR
GIT_LFS_SKIP_SMUDGE=1 uv sync
GIT_LFS_SKIP_SMUDGE=1 uv pip install -e .
These commands follow the repository README. The GIT_LFS_SKIP_SMUDGE=1 setting is required when fetching the LeRobot dependency, even if your immediate goal is simulated evaluation. Keep it in place.
The policy server requires Python 3.11 or newer, according to its pyproject.toml. The LIBERO evaluator inherits older Python 3.8 dependencies. Use separate environments instead of trying to reconcile their torch versions inside a single interpreter.
Keep the server terminal at the STAR repository root and start with native inference:
uv run scripts/serve_policy.py \
--env LIBERO \
--inference-mode NATIVE
The LIBERO environment selection loads configuration pi05_libero and the public checkpoint at gs://openpi-assets/checkpoints/pi05_libero. Weights are downloaded rather than bundled in the repository. First startup may include download and JAX compilation, so it does not represent steady-state latency.
Native and STAR evaluations must use the same weights, observation transforms and normalization statistics. Otherwise, a performance difference cannot be attributed specifically to residual steering.
5. Prepare the LIBERO-PRO evaluator and task assets
Open another terminal at the STAR root. The example below adapts the repository's evaluator instructions to install LIBERO-PRO, rather than only the original LIBERO submodule:
git clone https://github.com/Zxy-MLlab/LIBERO-PRO.git ../LIBERO-PRO
uv venv --python 3.8 examples/libero/.venv
source examples/libero/.venv/bin/activate
uv pip sync examples/libero/requirements.txt \
third_party/libero/requirements.txt \
--extra-index-url https://download.pytorch.org/whl/cu113 \
--index-strategy=unsafe-best-match
uv pip install -e packages/openpi-client
uv pip install --no-deps -e ../LIBERO-PRO
export PYTHONPATH="$PWD/../LIBERO-PRO${PYTHONPATH:+:$PYTHONPATH}"
This adapted recipe retains STAR's evaluator dependencies, then installs LIBERO-PRO code with --no-deps. LIBERO-PRO's requirements pin robosuite==1.4.0, whereas STAR's evaluator pins 1.4.1; directly combining those requirements would conflict. Treat this as an integration step to verify with the smoke test, rather than an environment we have benchmarked ourselves. Do not install the older simulator torch dependencies into the policy server environment. Compare the STAR evaluator instructions with the LIBERO-PRO installation guide when resolving problems. The two processes communicate through WebSocket and do not need identical torch versions.
Download the bddl_files and init_files assets from the official LIBERO-PRO dataset. BDDL files describe tasks and success conditions; initialization files provide starting simulator states. Following the upstream layout, place them under the LIBERO-PRO checkout's libero/libero/bddl_files/ and libero/libero/init_files/ directories.
For example, if the download is already extracted into libero_data at the STAR root:
cp -a libero_data/bddl_files/. ../LIBERO-PRO/libero/libero/bddl_files/
cp -a libero_data/init_files/. ../LIBERO-PRO/libero/libero/init_files/
Check LIBERO's configured data paths if the first import prompts you to configure them. They must refer to the intended LIBERO-PRO checkout. Next, confirm the package location and suite registration:
python - <<'PY'
import libero
from libero.libero import benchmark
print("Package:", libero.__file__)
required = {
"libero_10_lan", "libero_10_object", "libero_10_swap",
"libero_10_task", "libero_10_env",
}
registered = benchmark.get_benchmark_dict()
missing = required.difference(registered)
assert not missing, f"Missing suites: {sorted(missing)}"
print("All five LIBERO-PRO suites are registered")
PY
Registration verifies only the Python registry. A rollout can still fail if task definitions or initial states are missing. The next step exercises those additional parts of the pipeline.
6. Run a smoke test and enable STAR
With the native server listening on port 8000, run this command inside the evaluator environment:
MUJOCO_GL=egl python examples/libero/main.py \
--args.task-suite-name libero_10_lan \
--args.host 127.0.0.1 \
--args.port 8000 \
--args.env-control-hz 10 \
--args.replan-steps 5 \
--args.max-tasks 1 \
--args.num-trials-per-task 1 \
--args.video-out-path data/libero/native-smoke
One episode checks imports, rendering, checkpoint loading and network communication. Its outcome does not establish which method performs better. If EGL rendering fails, consult the repository's GLX alternative; GLX requires an appropriate display and is not a universal fix for headless machines.
Stop the native server with Ctrl+C. In the server terminal, launch STAR with the README configuration:
uv run scripts/serve_policy.py \
--env LIBERO \
--inference-mode STAR \
--star.beta-min 0.25 \
--star.beta-max 0.55 \
--star.beta-kappa 4.25 \
--star.phase-coeff-c 1.0 \
--star.phase-coeff-v 1.0 \
--star.phase-coeff-j -0.375 \
--star.phase-coeff-w -2.0 \
--star.correction-last-n-layers 6 \
--star-env-control-hz 10 \
--star-replan-steps 5
Repeat the evaluator smoke test with data/libero/star-smoke as its output directory. Merely passing --inference-mode STAR does not specify the entire README recipe: some script defaults differ from the explicit values above.
The environment operates at 10 Hz and the policy replans every five actions, giving a nominal inference request frequency of 2 Hz before computational delays. The client sends executed state history so the scheduler can observe motion between replanning events. Keep the server and evaluator frequency settings consistent. Also preserve the evaluator's policy reset at each episode boundary so one trial's history cannot contaminate the next.
To load a compatible local checkpoint, the README supports the policy:checkpoint subcommand with --policy.config pi05_libero and --policy.dir /path/to/checkpoint. That directory needs compatible params/ and assets/ trees. Model parameters alone are insufficient if normalization assets are missing.
7. Training: what is required and what is optional?
There is no STAR training stage in the workflow above. No optimizer or backward pass updates model weights. You are modifying the forward computation of an already trained policy. Training-free therefore describes the STAR intervention; it does not mean the underlying π0.5 checkpoint has never been trained or adapted.
For the paper's real-robot pear-to-bowl task, the authors collected 100 episodes and fine-tuned π0.5 for 15,000 steps. Native inference and STAR then used that same adapted checkpoint. The fine-tuning prepared the underlying policy for the task, while STAR remained an inference-only modification.
The public release currently excludes the real-robot, OpenVLA, GR00T-N1.5 and SimplerEnv implementations. The LIBERO commands above therefore cannot reproduce all of those experiments. Avoid treating an available result table as evidence that the corresponding deployment stack has been released.
For your own robot, first establish a working native pipeline with correct observations, action conventions and a suitable checkpoint. Evaluate STAR afterward. Residual steering does not replace demonstrations or task adaptation when the base policy has not acquired the necessary behavior.
8. Results: distinguish the website from paper v1
The repository README and project website report π0.5 improving from 48.2% to 50.4%, a gain of 2.2 percentage points. The project's detailed split results are:
| LIBERO-PRO split | Native | STAR on project website |
|---|---|---|
| Language | 92.8% | 92.8% |
| Object | 66.2% | 68.6% |
| Swap | 9.2% | 13.0% |
| Task | 6.8% | 7.6% |
| Environment | 65.8% | 70.2% |
| Average | 48.2% | 50.4% |
However, Table 2 in the arXiv v1 HTML lists STAR split scores of 92.6 / 67.0 / 7.2 / 18.8 / 65.0, averaging 50.1%. Appendix B also specifies different π0.5 coefficients: c=0.25, v=1.0, j=-0.125, w=-0.25, kappa=2.30. The README uses 1.0, 1.0, -0.375, -2.0, 4.25, respectively. Both use beta bounds of 0.25–0.55.
The public information checked here does not establish the cause of these differences. We therefore use the README recipe for the practical commands and identify the two sets of reported results separately. Record your exact code revision, configuration source and runtime arguments when evaluating; “STAR paper settings” alone is ambiguous.
The project also examines input-gradient saliency to measure how object pixels influence final actions. Attention maps can remain similar despite changes to hidden states, so an unchanged attention heatmap does not demonstrate that STAR has no effect. Conversely, stronger object saliency does not guarantee improved success on every individual task.
Other author-reported results include 40.3% → 43.2% for GR00T-N1.5 on LIBERO-PRO, 41.0% → 46.3% for GR00T-N1.5 on SimplerEnv and 30.3% → 36.0% for OpenVLA on SimplerEnv. The real-robot experiment improves from 17/40 to 19/40 successes. Its 3.3% inference latency overhead was measured on an RTX 3090 real-robot system, and should not be generalized to every LIBERO setup.
9. Evaluate fairly and troubleshoot systematically
After smoke tests, remove --args.max-tasks 1, set --args.num-trials-per-task 50 and --args.seed 7, then evaluate all five suites. The π0.5 protocol uses ten tasks per split and fifty episodes per task: 2,500 episodes per method, or 5,000 for a complete native-versus-STAR comparison.
Hold initial states, image preprocessing, action execution horizon and rendering conditions constant. Use separate video directories for each suite and mode. Inspect logs to distinguish genuine task failure from an exception that interrupted an episode. A partially completed batch is not interchangeable with the reported full benchmark.
For connection errors, confirm that server initialization has finished and that host and port agree. For missing suites, inspect libero.__file__ to detect an unintended original LIBERO installation. For missing task assets, inspect the dataset layout and configured paths. For memory failures, check other GPU processes and the actual checkpoint footprint before changing algorithm parameters.
Finally, STAR requires access to intermediate transformer states; a black-box policy API exposing only actions is insufficient. Its present proprioceptive cues focus on a single end-effector. Extending them to bimanual or whole-body control requires additional investigation. A useful first result is a controlled comparison on the released π0.5 implementation, with a clearly identified configuration and enough logs to explain what actually ran.
