VLA-Precision: ACoB Online RL in 45 Minutes

What VLA-Precision Solves

VLA-Precision is a real-world online reinforcement learning framework for Vision-Language-Action models, introduced in VLA-Precision: Asymmetric Co-Bootstrapping for Efficient Real-World Online RL of Vision-Language-Action Models by Chenyu Su and co-authors. The project page is available at vla-precision.github.io, and the implementation is open sourced in scy-v/VLA-Precision. The headline result is strong: across nine high-precision chemistry manipulation tasks, the system reaches 98.3% average success after 45.8 minutes of online training per task, with successful episodes taking 27.6 seconds on average.

If you have read our guides on EXPO-FT, TORL for tactile VLA manipulation, or reward models in LeRobot v0.6, the pattern is familiar. Imitation learning can make a robot competent. Online RL is often what makes it reliable. VLA-Precision targets this reliability gap directly. It does not train a robot policy from scratch. It starts from a $\pi_{0.5}$ model already fine-tuned on demonstrations, freezes most of the large VLA, and improves the action expert through ACoB, human corrections, real rollouts, and progressively calibrated value estimates.

The target domain is harder than a clean pick-and-place benchmark. Attaching a pipette tip, transferring transparent cuvettes, moving 2 mL vials into a rack, inserting rubber stoppers, extinguishing an alcohol lamp, and brushing a test tube with two robot arms are all precision tasks. A few millimeters of error can jam a part, break glassware, push with the wrong force, miss a hole, or require manual recovery. A behavior-cloned VLA may look impressive at 60-80% success, but that is still too fragile for repeated lab operation. Online RL can optimize real rewards, but early critics are unreliable, large VLAs are expensive to update, and unsafe exploration is unacceptable on physical hardware.

VLA-Precision contributes two linked ideas. The first is ACoB, or Asymmetric Co-Bootstrapping, an algorithm that combines fast behavior learning from interventions with slower but increasingly reliable value calibration. The second is ACoB-Stream, an actor-learner system architecture that makes online RL practical for large VLAs by caching context, streaming only the required state, and synchronizing only the trainable action-expert parameters.

The Core Idea Behind ACoB

Consider a robot attaching a pipette tip. The VLA proposes an action chunk: move down, align the pipette, and press with enough axial force. If the action drifts sideways, the operator takes over with a keyboard or an isomorphic master arm. The episode may still finish successfully after the correction, but the data contains three different pieces of information:

The original policy proposal may be wrong at this state.
The human-corrected action should be ranked higher.
The final episode outcome gives long-horizon success or failure.

A standard temporal-difference critic observes the executed action, not the overwritten proposal. If the episode succeeds after human correction, the critic may still leave the original failed proposal overvalued because that proposal never received its own transition. Later, if the actor directly maximizes absolute Q values, this error can create policy drift: the VLA follows a spurious high-value action instead of making a reliable local improvement.

ACoB fixes this with three signals.

First, global return propagation uses TD learning to propagate long-horizon reward through executed trajectories. Each transition contains the state, the action chunk actually executed by the robot, the reward inside the chunk, the next state, and a terminal flag. An ensemble of critics learns value estimates grounded in real robot outcomes.

Second, local preference ranking turns interventions into state-matched comparisons. When a human correction is meaningfully different from the policy proposal, ACoB enforces a ranking margin: the corrected action should have higher advantage than the original proposal at the same state. This is more informative than simply adding the corrected action to a behavior-cloning dataset.

Third, relative-advantage policy improvement avoids direct Q maximization. The current action expert and the frozen reference action expert decode action chunks from the same context and the same noise sample. ACoB compares their advantages within each critic and then takes a pessimistic minimum across the ensemble. The actor receives a strong improvement signal only when the critics agree that the current action is better than its baseline. This reduces sensitivity to Q scale and over-optimistic critic errors.

ACoB still uses behavior cloning, but it uses it as an active learning signal rather than only as a constraint. Its flow-matching behavior cloning objective trains on demonstrations, successful online executions, and effective human corrections. Early in training, this lets the action expert quickly absorb the best available actions. As the online data improves, the critic receives better trajectories and better intervention pairs. As the critic becomes better calibrated, relative-advantage updates can push the policy beyond the demonstration ceiling. That is the "asymmetric" part: fast behavior learning improves the data distribution, while slower value learning turns that data into stable autonomous improvement.

System Architecture

The complete pipeline has two stages:

Stage I: demonstrations -> full-parameter fine-tuning of pi0.5
Stage II: freeze the base VLA -> train LoRA in the action expert with ACoB online RL

In the paper formulation, the state includes visual observations, robot proprioception, and the language instruction. The VLA outputs an action chunk of horizon $H$, not a single low-level command. In OpenPI/pi0.5 terms, the frozen multimodal prefix encodes images, language, and state into a context $z_t$. A flow-based action expert then generates an action chunk from that context and Gaussian noise. In Stage II, VLA-Precision freezes the base prefix and action expert, adds trainable LoRA parameters to the action expert, and keeps the Stage-I LoRA state as a frozen reference for regularization.

ACoB-Stream is the systems layer that makes this algorithm run on real robots. The repository separates concerns cleanly. integrations/openpi handles the OpenPI boundary, acob owns the losses and critic networks, acob_stream owns actor/learner loops, buffers, communication, and checkpoints, and robotics owns robots, cameras, grippers, teleoperation, and environments. This matters when you extend the system: a new robot should implement the robot interface and register a factory, not modify the ACoB algorithm.

ACoB-Stream has four practical mechanisms:

This is why the paper spends so much attention on efficiency. For a compact policy, replay and model updates are manageable. For a large VLA, frozen-prefix compute, accumulated context memory, random disk access, and full-model synchronization can dominate the online loop. ACoB-Stream uses invariant-state decoupling: anything unchanged stays resident or cached; anything needed only by a specific objective is streamed on demand.

Installing the Repository

The repository uses uv and separates dependencies into three groups: Stage I on the GPU server, Stage II on the GPU server, and real-robot runtime on the robot machine. For a real deployment, clone the repository on both machines and keep the same task/deployment configuration pair across all participating processes.

curl -LsSf https://astral.sh/uv/install.sh | sh
git clone https://github.com/scy-v/vla-precision.git
cd vla-precision

# GPU server for Stage I
uv sync --frozen --group stage1

# GPU server for Stage II
uv sync --frozen --group stage2

# Real robot machine for control, cameras, and bridge
uv sync --frozen --group real-robot

If local storage is limited or slow, the README suggests redirecting training artifacts:

ln -s /path/to/storage/checkpoints ./checkpoints
ln -s /path/to/storage/train_data ./train_data

You will mainly edit three kinds of YAML:

The simple rule is: the task YAML says what the robot should learn; the deployment YAML says where and on which hardware it runs. Stage II resolves defaults, then task YAML, then deployment YAML, then command-line overrides.

Collecting Demonstrations

VLA-Precision supports demonstration collection in LeRobot format. The README points to UR5e/UR7e keyboard teleoperation, UR5e/UR7e isomorphic master-slave teleoperation, dual UR VR teleoperation, and Franka teleoperation. The paper uses two complementary intervention interfaces. For final alignment at millimeter or submillimeter scale, an incremental Cartesian keyboard is useful because it maps keypresses to fixed six-DoF translation or rotation increments. For longer sequences and compliant interaction, an isomorphic master arm gives the operator a more natural takeover path.

The state-action representation is deliberately practical. The state includes TCP pose relative to the start of the episode, TCP twist, measured force, measured torque, and gripper state. The action is a step-wise delta task-space command: six end-effector delta dimensions plus a gripper command. Demonstrations and interventions are recorded at a nominal 15 Hz. Single-arm setups use one wrist camera and one external camera. The dual-arm setup uses two wrist cameras and one external camera, giving both arm-local and shared scene views.

For a new task, start with a narrow setup:

1. Define one stable language instruction, such as attach the pipette tip.
2. Define a reset procedure with a safe nominal pose and limited randomization.
3. Define a completion detector, either from sensors, a vision rule, or an operator signal.
4. Define a reward function with a clear terminal success reward and a small time reward or penalty.
5. Make sure task.image_keys, cameras.devices, and data.image_key_map use the same logical camera keys.

Stage I: Fine-Tune OpenPI/pi0.5

Stage I creates the starting policy for online RL. This is not a minor step. ACoB assumes the VLA already has enough task competence to generate useful rollouts and useful intervention opportunities. First compute normalization statistics, then run full-parameter OpenPI fine-tuning.

# GPU server
uv run --no-sync main.py \
  --stage stage1 \
  --mode norm-stats \
  --config configs/stage1/insert_two_bottles_diagonal_rack.yaml

uv run --no-sync main.py \
  --stage stage1 \
  --mode train \
  --config configs/stage1/insert_two_bottles_diagonal_rack.yaml

The paper reports that VLA-Precision often uses fewer Stage-I demonstrations than the pure VLA fine-tuning baselines. For pipette tip attachment, VLA-Precision uses 60 Stage-I demos and 5,000 steps, while the $\pi_0/\pi_{0.5}$ baselines use 120 demos and 25,000 SFT steps. For tube brushing and pipette transfer/ejection, it uses 120 demos and 15,000 Stage-I steps. The goal is not to make Stage I perfect. The goal is to create a strong prior that Stage II can improve without destroying pretrained competence.

Stage II: Run ACoB Online RL

Before online RL, preprocess part of the offline data so the replay buffer and context buffer are initialized. This step materializes action chunks, transitions, and OpenPI contexts, which is important for large-VLA throughput.

# GPU server
uv run --no-sync main.py \
  --stage stage2 \
  --mode preprocess \
  --config configs/stage2/tasks/insert_two_bottles_diagonal_rack.yaml \
  --deployment configs/stage2/deployments/single_ur.yaml

Then start four processes with the same task and deployment YAMLs:

Robot machine: serve-robot
Robot machine: robot-agent-bridge
GPU server: learner
GPU server: actor

On the robot machine:

uv run --no-sync main.py --stage stage2 --mode serve-robot \
  --config configs/stage2/tasks/insert_two_bottles_diagonal_rack.yaml \
  --deployment configs/stage2/deployments/single_ur.yaml

uv run --no-sync main.py --stage stage2 --mode robot-agent-bridge \
  --config configs/stage2/tasks/insert_two_bottles_diagonal_rack.yaml \
  --deployment configs/stage2/deployments/single_ur.yaml

On the GPU server:

uv run --no-sync main.py --stage stage2 --mode train --role learner \
  --config configs/stage2/tasks/insert_two_bottles_diagonal_rack.yaml \
  --deployment configs/stage2/deployments/single_ur.yaml

uv run --no-sync main.py --stage stage2 --mode train --role actor \
  --config configs/stage2/tasks/insert_two_bottles_diagonal_rack.yaml \
  --deployment configs/stage2/deployments/single_ur.yaml

The learner waits until the replay and correction buffers are ready, warms up the critic, and then loops. It samples half a batch from replay and half from corrections, retrieves the required contexts, updates the critic several times, updates the LoRA action expert once, and publishes the new trainable state periodically. The actor keeps collecting real rollouts with the latest valid policy state, while the robot-side bridge handles hardware control and human intervention.

Inference and Evaluation

Once you have a Stage II checkpoint, evaluate it with --mode evaluate. The evaluation model can run either on the GPU server or locally on the robot machine. If it runs locally, install the matching dependency group for the model.

# Robot machine: keep robot service and bridge running
uv run --no-sync main.py --stage stage2 --mode serve-robot \
  --config configs/stage2/tasks/insert_two_bottles_diagonal_rack.yaml \
  --deployment configs/stage2/deployments/single_ur.yaml

uv run --no-sync main.py --stage stage2 --mode robot-agent-bridge \
  --config configs/stage2/tasks/insert_two_bottles_diagonal_rack.yaml \
  --deployment configs/stage2/deployments/single_ur.yaml

# Stage II ACoB model
uv run --no-sync main.py --stage stage2 --mode evaluate \
  --config configs/stage2/tasks/insert_two_bottles_diagonal_rack.yaml \
  --deployment configs/stage2/deployments/single_ur.yaml

Results are written after every episode under results/<experiment>/acob/<time>.json. For a serious lab deployment, do not track success alone. Also watch intervention rate, average episode time, timeouts, dropped camera frames, force spikes, recovery behavior, and reset failures. A policy that reaches 100% success but repeatedly hits high force during insertion is not ready for unattended operation.

Reported Results

The evaluation covers four task categories: contact-rich, contact-light, contact-free, and bimanual coordination. The hardware includes a UR5e with a PGI gripper, a UR5e with a LinkerHand dexterous hand, two UR5e arms with PGI grippers, and a Franka Research 3 with a PGI gripper. Each task includes pose randomization rather than testing a single fixed setup.

The key numbers are:

Against Robo-Dopamine, the strongest real-world RL baseline in the table, VLA-Precision improves average success by 88.3 percentage points and execution speed by 61.2%. The ablations are also revealing. Full ACoB reaches 96.25% final autonomous success on four representative tasks. Removing critic preference drops it to 26.25%. Removing actor behavior cloning drops it to 20.00%. Replacing relative advantage with direct Q maximization drops it to 8.75%. These results support the paper's argument that all three components are necessary: preference ranking fixes the local credit ambiguity, actor BC rapidly absorbs useful corrections, and relative advantage suppresses value-induced policy drift.

The systems results matter just as much. Disk ACoB-Stream completes 15,666 critic-to-actor cycles in 150 minutes, reaching 1.7407 CTA/s and 0.574 seconds of mean cycle latency. Compared with the No KV Cache ablation, it provides 10.95 times the throughput and reduces mean latency by 90.9%. Full-history random sampling begins to degrade after roughly 43 minutes as the replay working set exceeds Linux page-cache capacity, while the sliding-window disk-backed design keeps throughput stable.

A Practical Beginner Path

Do not start with the hardest bimanual task. A safer path is:

1. Choose a single-arm task with a clear completion detector, such as inserting an object into a slot or attaching a tip.
2. Collect clean demonstrations with moderate pose diversity and stable lighting.
3. Run Stage I until the policy has enough starting competence to produce partially useful autonomous rollouts.
4. Enable safe intervention with conservative force, velocity, and workspace limits.
5. Run short Stage II sessions first and check whether intervention rate decreases.
6. Evaluate on held-out object poses, not only on the same reset distribution used during online training.

If you are building a custom LeRobot pipeline, treat VLA-Precision as a systems blueprint rather than a single command to copy. The reusable ideas are a separate correction buffer, preference ranking for overwritten proposals, frozen-reference regularization, cached VLA context, and trainable-subspace policy synchronization. The hardest part is rarely writing the loss. The harder part is making the real robot produce stable data, reliable rewards, fast resets, and safe human takeovers.

Limitations

VLA-Precision is still task-specific online RL. The paper discusses multi-task real-world RL as an important future direction, but the main results optimize one task at a time. The compute requirement is also nontrivial: the shared training setup in the paper uses four NVIDIA A800 GPUs. Action horizon is another limitation. With step-wise delta actions, increasing the executed horizon from 3 to 6 reduces average autonomous success on three precision-insertion tasks from 58.3% to 6.7% and increases relative action-prediction RMSE by 29.3%. Longer chunks are not automatically better for precision contact.

The most useful lesson is that VLA-Precision respects the realities of physical labs. It does not assume the critic is correct early. It does not assume demonstrations are perfect. It does not push the full VLA through the network at every update. It does not treat human intervention as only a failure-recovery tool. For beginners, that is the right mental model: reliable online RL for precise robots is a joint problem of learning signal, system latency, human takeover, and hardware safety.

Related Posts

• EXPO-FT: Online RL for π0.5 VLA
• TORL: Tactile Online RL for Contact Manipulation
• LeRobot v0.6: Reward Models and Closed-Loop RL