Proxy Policy Steering, or PPS, is a clean and useful idea for teams deploying VLA policies on real robots: keep the large base policy frozen, train two lightweight proxy policies, and use their calibrated difference to steer the base sampler at inference time. The paper Proxy Policy Steering by Chuanruo Ning, Tianrui Wang, Wei-Chiu Ma, and Kuan Fang appears at CoRL 2026. The official resources are the project page, arXiv paper 2609.09148, and the open-source repo TritiumR/pps.
The motivation is practical. A generalist robot policy such as π0.5 already contains broad manipulation priors: grasping, object contact, recovery from small mistakes, and a useful mapping from language and images to action chunks. But for a specific deployment task, the base policy often gets close without being reliable. It may approach the right object but miss the insertion, open the wrong part of a container, place an object near the target instead of inside it, or fail when timing matters. Fine-tuning the full base model or adding LoRA adapters can improve imitation on the demonstrations, but it can also overwrite useful priors that the small task dataset never exercises. PPS avoids that failure mode by never modifying the base VLA.
Why Two Proxy Policies?
The tempting simple version would be: train one small task policy on demonstrations, then use it to guide the base policy. PPS argues that this is not enough. A small policy trained from a small demonstration set contains two different things: the task-specific behavior we want, and the approximation errors of a smaller model. If we directly push the base policy toward that small policy, we may steer toward both.
PPS separates those factors with two proxies:
| Proxy | How it is trained | What it represents |
|---|---|---|
π_ref, the reference proxy |
Distilled from the frozen base on target-task observations | How the base behaves in the task-relevant state region |
π_task, the task proxy |
Initialized from π_ref, then trained on task demonstrations |
How task supervision changes that behavior |
The important design choice is that the two proxies share architecture, action representation, flow-matching schedule, and initialization path. The task proxy starts from the reference checkpoint. Therefore, the residual v_task - v_ref is much closer to the change induced by task supervision than to a generic small-model policy. This is the "proxy" part of PPS: the proxies do not replace the base; they estimate a calibrated direction for adapting it.
The inference rule is compact:
v_PPS(x, k, o, l) = v_base(x, k, o, l) + γ * (v_task(x, k, o) - v_ref(x, k, o))
Here x is the noisy action chunk at noise level k, o is the observation, l is the language instruction, and γ is the steering strength. In the official README, the implementation appears as:
v_t[:, :, :proxy_action_dim] += steer_scale * (task_v_t - ref_v_t)
If the base is heading toward a wrong action mode and the task proxy has learned a successful mode, the residual pulls the denoising trajectory toward the task behavior. If the state is outside the demonstrations, or if the two proxies agree, the residual becomes small and the frozen base prior remains in control. This is why PPS can preserve recovery behavior that LoRA fine-tuning may damage.
Architecture
PPS is designed for flow-matching VLAs. In these policies, the model does not expose a normalized action probability that is easy to edit directly. Instead, it predicts a velocity field that moves noise into an action chunk. π0 and π0.5 belong to this family: they take images, robot state, and language, then generate a short horizon of continuous robot actions through a denoising or flow integration process.
In the paper, the frozen base is the state-of-the-art π0.5 VLA, roughly a 3B-parameter model. The proxy policies are far smaller. The appendix describes a DINOv3 ViT-S/16 visual encoder followed by a small Gemma-style transformer action expert that regresses flow-matching velocity in the same action space. The key requirement is not that the proxy matches the base capacity; it is that the proxy and base use compatible action representation and flow schedule, so the residual can be added in velocity space.
The full inference path looks like this:
Observation + instruction
|
v
Frozen π0.5 base predicts v_base
|
+-----------------------------+
| |
v v
Reference proxy predicts v_ref Task proxy predicts v_task
| |
+-------------+---------------+
v
residual = v_task - v_ref
|
v
guided velocity = v_base + γ * residual
|
v
flow sampler outputs action chunk
|
v
robot executes the first actions, then replans
The paper reports a 15 Hz control frequency. Each policy produces an action chunk of length 15, and the system executes the first 8 actions before replanning. PPS applies the guided velocity at every denoising step, not as a post-processing correction after the final action is sampled. That matters because it changes the trajectory of the sampler itself. The base distribution can be nudged into a successful task mode before the final action chunk has collapsed.
Installing the Official Repo
The official TritiumR/pps repository is organized around simulation evaluation, OpenPI-based policy training, and a real-robot stack:
eval_steering.py steered rollouts with base + task proxy + reference proxy
eval_pi.py unsteered π0 / π0.5 rollouts
run_eval_chain.sh run several tasks back to back
task_prompts.json task ids, prompts, and checkpoint paths
openpi/ OpenPI fork: models, configs, train, distill, serve
IsaacLab/ bundled IsaacLab code and manipulation tasks
droid/ real-robot stack
media/teaser.gif README teaser GIF
Simulation evaluation requires NVIDIA Isaac Sim and IsaacLab dependencies, plus a suitable GPU. The repo imports the bundled IsaacLab/ directory, so you do not need a separate IsaacLab checkout. First install the Hugging Face helper used by the download scripts:
pip install huggingface_hub
Then fetch the scene assets and released proxy checkpoints from the repository root:
python IsaacLab/fetch_assets.py
python openpi/fetch_checkpoints.py
The assets are about 2.7 GB and the proxy checkpoints are about 0.8 GB. You can also fetch a subset:
python openpi/fetch_checkpoints.py --task pot weight
One small but important detail: checkpoints must live under a directory literally named checkpoints. The evaluation script derives the training config name from the path by finding the checkpoints segment and reading the next path segment. If you unpack files into an arbitrary directory, eval_steering.py can fail because it cannot derive the config.
The expected layout is:
openpi/checkpoints/
├── pytorch/pi05_droid_jointpos/ # base policy, supplied by you
├── proxy_isaaclab_droid_pot_pi05_jointpos/
│ ├── reference/20000/
│ └── task/24000/
├── proxy_isaaclab_droid_tea_pi05_jointpos/
│ ├── reference/20000/
│ └── task/32000/
└── proxy_isaaclab_droid_weight_pi05_jointpos/
├── reference/20000/
└── task/24000/
The base π0.5 checkpoint is not included in the release. You must place a trained pi05_droid_jointpos checkpoint at openpi/checkpoints/pytorch/pi05_droid_jointpos before running steering evaluation.
Training: Reference First, Task Second
PPS training has two stages. Both run from the openpi/ directory, and <train-config> is a registered proxy training config such as proxy_isaaclab_droid_pot_pi05_jointpos.
Stage 1 is on-policy reference distillation. The reference proxy watches the frozen base model on target-task observations and learns to match its velocity predictions along the generation trajectory:
cd openpi
uv run scripts/distill_pytorch.py proxy_isaaclab_droid_pot_pi05_jointpos \
--exp_name reference \
--teacher_checkpoint_dir checkpoints/pytorch/pi05_droid_jointpos
The output is typically:
checkpoints/proxy_isaaclab_droid_pot_pi05_jointpos/reference/20000
Stage 2 is task specialization. The task proxy starts from the reference checkpoint and is trained on task demonstrations:
uv run scripts/train_pytorch.py proxy_isaaclab_droid_pot_pi05_jointpos \
--exp_name task \
--pytorch_weight_path checkpoints/proxy_isaaclab_droid_pot_pi05_jointpos/reference/20000
The training run may produce checkpoints such as:
checkpoints/proxy_isaaclab_droid_pot_pi05_jointpos/task/8000
checkpoints/proxy_isaaclab_droid_pot_pi05_jointpos/task/16000
checkpoints/proxy_isaaclab_droid_pot_pi05_jointpos/task/24000
The paper uses 50 demonstrations per real-world task. For simulation, the demonstrations are generated with MimicGen from 5 teleoperated source demonstrations. This is a useful regime for robotics teams because it does not require thousands of trajectories, but it still requires high-quality data, correct action normalization, and task resets that match the evaluation protocol.
Inference and Evaluation
For a single steered simulation run, the README gives a command like this for the pot task:
CFG=openpi/checkpoints/proxy_isaaclab_droid_pot_pi05_jointpos
python eval_steering.py \
--task "Isaac-Pot-Droid-Visuomotor-v0" \
--exp_name eval_pps \
--base_checkpoint_dir openpi/checkpoints/pytorch/pi05_droid_jointpos \
--task_checkpoint_dir "$CFG/task/24000" \
--ref_checkpoint_dir "$CFG/reference/20000" \
--prompt "remove the lid of the pot and put egg in it" \
--steer_scale 0.4 \
--task_num_steps 1200
To evaluate the unsteered base policy, use eval_pi.py instead:
python eval_pi.py \
--task "Isaac-Pot-Droid-Visuomotor-v0" \
--model_name pi05_droid_jointpos \
--checkpoint_dir openpi/checkpoints/pytorch/pi05_droid_jointpos \
--prompt "remove the lid of the pot and put egg in it" \
--name eval_pi05 \
--max_steps 1200
The helper script run_eval_chain.sh reads task_prompts.json and can run the released tasks in sequence:
./run_eval_chain.sh
./run_eval_chain.sh pot weight
./run_eval_chain.sh --list
The default steering scale is 0.4. The paper sweeps γ and finds that the best region is typically around 0.4 to 0.6 in the 50-demonstration setting. If γ is too small, PPS behaves like the base model. If it is too large, the proxies dominate and the system loses some of the base prior that makes the method robust.
Main Results
PPS is evaluated on 8 real-world and 4 simulation manipulation tasks. The tasks include long-horizon goals, articulated objects, deformable objects, and out-of-distribution settings where the base policy does not solve the task by itself. The real-world setup follows the DROID-style robot configuration with one third-person camera and one wrist camera. The paper reports 10 rollouts per task per method in the real world, and 100 simulation rollouts per task per method with randomized initial configurations.
The headline numbers are strong:
| Evaluation group | Base | PPS | Interpretation |
|---|---|---|---|
| π0.5 real-world average | 22% | 79% | +57 points across 8 tasks |
| π0.5 simulation average | 12% | 64% | +52 points across 4 tasks |
| π0 real-world transfer | 5% | 55% | The recipe transfers beyond π0.5 |
The paper summarizes the gain as roughly 55% absolute success rate for π0.5 on average, with zero-to-one improvements on tasks the base never solves. PPS also outperforms LoRA fine-tuning, from-scratch specialists, residual policies, and earlier inference-time steering baselines.
The preservation of base capability is just as important as the final success rate. Under mid-rollout disturbances, PPS loses only 7.8 success-rate points, while LoRA loses 15.8 points. A PPS variant without the reference proxy loses 26.1 points. This supports the central claim: the reference proxy helps cancel proxy-specific bias, so the residual behaves like a task correction rather than a replacement policy.
What the Ablations Teach
Three ablations are especially useful if you want to reproduce PPS.
First, without the reference proxy, the residual is no longer a clean task-induced change. The task proxy carries approximation errors, and the base policy does not share those errors. The result is a weaker and less robust steering signal.
Second, without velocity-level distillation, the reference proxy is supervised in the wrong space. PPS steers a flow-matching sampler at multiple noise levels, so it needs a velocity field that matches the base along the relevant denoising path.
Third, without reference initialization, the task proxy is trained from scratch. Then π_task and π_ref differ because of random initialization and optimization history, not only because of task supervision. The residual becomes noisier and less meaningful.
When Should You Use PPS?
PPS is a good fit when you already have a strong flow-matching VLA, want to specialize it to a new manipulation task, and do not want to modify the base weights. A typical scenario is a lab or robotics team with a π0.5-style checkpoint that performs broad manipulation reasonably well, but needs a narrow task to become reliable. You can collect 50 to 200 good demonstrations, train two small proxies, and preserve the frozen base policy for recovery and broad priors.
PPS is not a universal shortcut. The paper focuses on a single-task regime. The base and proxies need compatible action representations and schedules. If the base has no useful prior for the task at all, a from-scratch specialist or a different base model may be better. The current repo release also includes proxy checkpoints only for some simulation tasks such as pot, tea, and weight; capsule appears in the JSON, but its checkpoints are not part of the release.
For beginners, a good learning path is:
1. Run eval_pi.py first to measure the unsteered base.
2. Run eval_steering.py with steer_scale = 0.4.
3. Sweep 0.2, 0.4, 0.6, and 0.8.
4. Watch rollout videos, not only success numbers.
5. Check whether PPS recovers from object shifts and contact mistakes.
6. If it only works on demo-like states, improve reference distillation or collect broader demos.
The larger message is that VLA adaptation does not always need to be weight adaptation. If the base policy already contains broad manipulation knowledge, it can be better to keep that knowledge untouched and learn a calibrated steering direction around it. PPS turns two small proxy policies into a practical inference-time adapter for π0.5-style manipulation.
