πR² + GR00T-N1.7: Real-Time VLA Control

πR², pronounced "pi R squared", is a practical method for turning a large VLA such as GR00T-N1.7 into a real-time reactive manipulation policy. The problem it targets is very concrete on physical robots: modern VLA policies usually generate action chunks with diffusion or flow matching. They are powerful because they use a large vision-language backbone, but each inference call is expensive. While the model computes the next chunk, the robot keeps executing an older chunk almost open-loop. If an object slips, the hand touches at the wrong angle, a book falls, or contact force changes quickly, the policy may react too late.

πR² does not try to replace GR00T-N1.7 with a small model. It keeps the large backbone, keeps action chunking, and changes how conditioning and denoising are scheduled so the robot can use fresh signals at every control tick. In the original paper, when fine-tuned from GR00T-N1.7 on an xArm6 + XHand platform, πR² replans at roughly 25 Hz on an RTX A5000, meaning it can emit actions using fresh proprioception every 40 ms. The base pipeline replans at roughly 7 Hz because it has to pay the full vision-language and multi-step denoising cost. Across simulation and real-world manipulation tasks, the authors report success-rate gains of up to 23% in simulation and 30% in the real world over the strongest baseline.

This guide is for readers who know the basics of imitation learning or VLA, but do not need to be diffusion experts. The goal is to help you read the paper, understand the official repository, install the stack, fine-tune the pir2 variant, and decide when πR² is the right upgrade over a standard GR00T-N1.7 deployment.

Original project and problem statement

The original paper is πR²: Reactive Real-time Flow Policies, by Sungjae Park and Shubham Tulsiani from Carnegie Mellon University. The project page states the core idea clearly: πR² makes large-scale action-chunking flow policies such as VLAs reactive and real-time using a fast proprioception channel, an asynchronous vision-language channel, and a latency-adaptive flow schedule. The official code lives in pi-r2-flow/pi-r2-flow; the training code is a fork of NVIDIA Isaac-GR00T pinned to the pir2 branch, adding πR² to the flow-matching action head of GR00T-N1.7.

To understand why this matters, start from a normal VLA pipeline. The robot receives camera images, a language prompt, and robot state. The vision-language backbone extracts semantic features, then the action head uses several denoising steps to produce a chunk of future actions. Chunks improve temporal consistency and training stability, but they create a cost: the robot commits to actions predicted from old observations. If the control loop runs at 25 Hz but inference only updates at about 7 Hz, the robot may pass through 3-4 control ticks before the model looks at the latest state again.

In robot manipulation, 40-160 ms can be a long time. A ball can roll, a book can slide out of the fingers, a tilted object can fall, or a force spike can appear and disappear. The VLA may understand the instruction correctly, but if contact feedback arrives too late in the action-generation loop, the manipulation still fails. πR² frames this as two coupled failures: low reactivity, because action chunks run open-loop, and high latency, because each action depends on stale observations.

Architecture: keep GR00T, change the reflex loop

GR00T-N1.7 is NVIDIA's 3B-parameter VLA model. According to Isaac-GR00T and NVIDIA's Hugging Face article, it uses an Action Cascade architecture: System 2 is a vision-language module based on Cosmos-Reason2-2B / Qwen3-VL that processes image tokens and language instructions; System 1 is a Diffusion Transformer action head that takes semantic features and robot state, then denoises them into continuous robot actions. Inputs include RGB frames, text instructions, and proprioceptive state such as joint positions, velocities, or end-effector poses. Outputs are continuous action vectors defined by the robot embodiment.

πR² does not replace this full stack. It uses an asymmetry already present in VLA systems: vision-language processing is expensive, while proprioception is cheap. Images and text must pass through a large backbone and often cost tens of milliseconds. The paper uses GR00T-N1.7 compute as the reference: vision/text VLM processing takes about 60 ms, and four denoising steps take about 80 ms. In contrast, joint state, torque, and fingertip force can be read continuously from the robot and processed much faster. πR² therefore splits conditioning into two channels:

The key is that πR² accepts stale vision-language features instead of forcing the whole pipeline to be synchronous. Vision provides coarse context: which object to pick, where the basket is, what the final goal should be. Proprioception provides local state: whether the finger has touched, whether the object is slipping, how far the joint is from the intended pose, and whether force is spiking. For reactive manipulation, many short-term corrections can be driven by fresh proprioception as long as the semantic plan remains valid.

The second ingredient is per-position diffusion forcing inside the action chunk. Standard diffusion or flow policies often denoise the whole chunk under one shared noise level. Diffusion forcing lets each action position in the buffer have its own noise level. πR² uses this to build a staircase schedule: the front of the buffer contains in-flight actions that are clamped as inpainting conditioning; the middle ramps from clean to noisy; the tail contains newly appended noisy slots for future actions. When measured inference latency is d control ticks, the schedule reshapes itself so one call performs one denoising step and emits the next d clean actions.

As a system diagram, the loop looks like this:

camera + language ──> slow VLM worker ──> cached slow features
       robot state ──> fresh fast channel at every tick

cached slow features + fresh proprioception + action buffer
                         |
                         v
             πR² flow action head, 1 NFE/call
                         |
                         v
        emit action(s), slide buffer, repeat at 25 Hz

With standard GR00T-N1.7, each policy call usually pays both the VLM cost and multiple denoising steps. With πR², the VLM can run asynchronously on a separate worker or GPU, while the action head uses the latest slow-feature cache and current proprioception. This is why the repository supports a 2-GPU split: one VLM server on port 5555, one DiT/action server on port 5556, and --async-vlm for the πR² deployment path.

Hardware and repositories

The official repository deploys on a dexterous manipulation setup:

If you do not have the same hardware, the repository is still useful for learning the policy structure and fine-tuning path. To reproduce the real-world results, however, you need an arm, a dexterous hand, a camera, a robot machine, and a GPU machine. The paper uses RTX A5000 hardware for real-world training and inference; its hyperparameter table reports batch size 512, bf16 + tf32, fused AdamW, peak learning rate 1e-4, weight decay 1e-5, gradient clipping at 1.0, and 8 A5000/A6000 GPUs for training. This is not a laptop-scale setup.

For beginners, the main point is simple: πR² is a method for a pretrained large flow policy. It is not a PID controller, not a teleoperation script, and not prompt engineering for a VLA. You need a demonstration dataset in GR00T/LeRobot format, a correct modality configuration for your state/action layout, and a GR00T-N1.7-3B base checkpoint.

Installation

The official README starts by cloning the repository with submodules, because the Isaac-GR00T fork is included under learning/:

git clone --recursive https://github.com/pi-r2-flow/pi-r2-flow.git
cd pi-r2-flow

If you already cloned without submodules:

git submodule update --init --recursive

The robot-side deployment stack lives under deployment:

cd deployment
pip install -e .
pip install -e ".[camera]"

Depending on your hardware, you will also need the RealSense driver, the xArm6 control library, RS485 setup for XHand, serial/USB permissions, and a stable network path between the robot machine and the GPU server. On a real robot, validate each layer before running the VLA:

1. The camera server returns frames with the expected resolution and timestamps.
2. The arm accepts slow joint commands safely.
3. The hand accepts 12 joint commands without reversed signs or wrong scaling.
4. Robot state logging is synchronized with action logging.
5. The emergency stop works independently of the policy process.

The training code is in:

cd learning/Isaac-GR00T

The README describes this directory as a fork of NVIDIA Isaac-GR00T pinned to the pir2 branch, adding πR² to the GR00T-N1.7 flow-matching action head. You install it using the Isaac-GR00T instructions, download the GR00T-N1.7-3B base checkpoint, prepare a dataset in GR00T/LeRobot format, and write a modality_config.py for your embodiment.

Dataset and modality config

Training uses the standard GR00T LeRobot format. Each episode needs video observations, robot state, actions, and task text. For xArm6 + XHand, the action vector has 18 dimensions:

action[0:6]   = xArm6 joints
action[6:18]  = XHand joints

The state should at least include current arm and hand joint positions. If you also have velocity, torque, or tactile/fingertip force, those signals fit πR² especially well because the fast channel is designed for fresh reactive data at every tick. The modality configuration must be exact: dataset keys, normalization statistics, dimensions, and joint order must all match what the model and robot driver expect.

A common failure is successful training followed by bad deployment because the action order is wrong. For example, the dataset may store XHand joints as thumb-index-middle..., while the driver expects index-middle-thumb.... The policy then appears to have learned nothing, even though the real bug is a vector mapping mismatch. For VLA action policies, vector order is an API contract. Before collecting large data, write a tiny test that sends a slow one-hot action and confirms which physical joint moves.

The real-world paper uses four tasks:

These tasks are not chosen only for semantic manipulation. They force the policy to react during contact. Catch Book is the clearest example: if the policy waits for a slow vision-language pipeline, the book has already slipped; if it uses fresh proprioception, the fingers can adjust grip almost immediately after force changes.

Fine-tuning the πR² variant

The repository uses the same launch_finetune.py entrypoint for three variants: standard flow, Train-Time RTC, and PI-R2. The README gives this command pattern:

cd learning/Isaac-GR00T

export GR00T_IMAGE_DELAY_MAX=5

torchrun --nproc_per_node=8 gr00t/experiment/launch_finetune.py \
  --base-model-path      <path/to/GR00T-N1.7-3B> \
  --dataset-path         <path/to/lerobot_dataset> \
  --modality-config-path <path/to/modality_config.py> \
  --embodiment-tag       NEW_EMBODIMENT \
  --num-gpus             8 \
  --global-batch-size    512 \
  --max-steps            40000 \
  --output-dir           <path/to/output> \
  --streaming \
  --streaming-constant-weight 0.2 \
  --streaming-chunk-wise-weight 0.8 \
  --streaming-schedule-mode pir2 \
  --streaming-chunk-size-max 5 \
  --streaming-mask-clean-end \
  --image-delay-max 5 \
  --image-delay-embed-dim 64

The important flags are:

According to the appendix, the real-world πR² fine-tune freezes the VLM backbone and vision encoder, and fine-tunes the action head: projectors plus DiT. Per-position AdaLN parameters are initialized from the pretrained shared pair, so each position in the action horizon starts from a uniform schedule and specializes during fine-tuning. The defaults include chunk length H=50, observation history n_obs=1, control rate 25 Hz, and 1 NFE per call for πR², while Flow and Train-Time RTC baselines use 4 NFE.

Training must simulate deployment latency. Each batch samples a delay d, builds a staircase schedule, clamps the first d slots as ground-truth in-flight actions, and computes loss only on the remaining slots. In real-world training, the slow channel is additionally delayed by a random d_vis from 0 to d_vis_max: the model receives an older image frame from the demonstration, while proprioception remains current. This is why deployment can tolerate a VLM feature cache that is 1-5 ticks stale.

Running inference on the robot

After fine-tuning, start the GR00T inference server on the GPU machine. The README provides a single-GPU baseline path:

# Single GPU for baselines
python gr00t/eval/run_gr00t_server.py \
  --model-path <checkpoint> \
  --port 5555 \
  --host 0.0.0.0

For πR² with asynchronous VLM, the repository uses a 2-GPU split:

MODEL_PATH=<checkpoint> bash scripts/gr00t_inference_2gpu.sh

If the GPU server is inside a cluster, forward the ports back to the robot machine:

ssh -N -L 5555:<compute-node>:5555 <username>@<cluster-login>
ssh -N -L 5556:<compute-node>:5556 <username>@<cluster-login>

Then start the camera server on the robot machine:

python deployment/apps/run_camera_server.py --port 5000

The repository supports three query modes:

For πR², the target is continuous or an equivalent pipelined setup plus asynchronous VLM. The action worker queries back-to-back; a new chunk replaces the active one at a swap boundary; actions already being executed are passed into the model as inpainting conditioning. πR² also adds a VLM-cache worker that refreshes visual features in the background. This is the difference from naive async: naive async computes a future chunk while the old chunk executes, but it does not correctly condition on in-flight actions; πR² knows which part of the buffer has already been committed and uses it as model input.

Results and how to read them

In simulation, the paper studies VLA-level inference delay using GR00T-N1.7 compute costs: about 60 ms for VLM/text processing and about 80 ms for four denoising steps. End-to-end baselines such as naive async and Train-Time RTC pay an effective delay of 1.75 d0; πR² without async pays 1.0 d0; full πR² with async holds proprioception delay at about 1 tick while visual delay grows separately. The simulation results show full πR² winning at every tested d0, with success around 0.43, 0.42, and 0.45, while naive async drops from 0.33 to 0.22, and Train-Time RTC drops from 0.36 to 0.19 as delay increases.

On the real xArm6 + XHand setup, the paper compares four methods, all fine-tuned from GR00T-N1.7 using the same data and budget:

The real-world table shows πR² leading on every task. A few key numbers:

πR² also improves progress scores across subgoals. That matters because manipulation is not just success or failure at the last frame. In Insert Box, for example, a policy may push and stand the box but fail during insertion; a progress score tells you whether the method learned intermediate phases. πR² improves both success and progress, which suggests it is not just getting lucky on a few trials.

When should you use πR²?

πR² is worth considering when your task has at least one of these properties:

If your task is almost quasi-static, the object state changes slowly, and you only need simple pick-and-place in a controlled scene, standard action chunking may be enough. πR² adds complexity: streaming code, delay schedules, cache workers, latency logging, and enough reactive data for the model to learn useful corrections. For a small team, start with a standard GR00T-N1.7 baseline, measure latency and failure modes, and only move to πR² if late reaction is the main failure.

Deployment checklist for a small lab

A practical rollout plan looks like this:

1. Run standard GR00T-N1.7 inference with a checkpoint fine-tuned for your robot.
2. Log all timestamps: camera frame, state read, policy request, policy response, and action send.
3. Measure the real control tick rate. Do not assume 25 Hz if your driver or network is stable only at 15-20 Hz.
4. Run sync, pipelined, and continuous baselines to identify failure modes.
5. Fine-tune plain_flow and rtc on the same dataset so comparisons are fair.
6. Fine-tune pir2 with image-delay-max close to your measured system latency.
7. During deployment, start at low speed, limit joint and hand commands, keep E-stop active, and increase speed only after action order is verified.

One easy mistake is focusing only on average FPS. πR² is about tail latency and effective delay. If your average is 40 ms but you occasionally spike to 200 ms, the robot may fail exactly during contact. Log a latency histogram, the delayed tick count d, and the frame age of the slow visual feature. When a failure happens, inspect how stale the visual feature was and whether proprioception was dropped or delayed.

Conclusion

πR² matters because it does not reject the value of large VLAs. Instead of saying "large models are too slow, use a smaller model", it asks a better systems question: if vision-language reasoning is slow but proprioception is fast, can we organize inference so the robot remains reactive? The answer is yes: split fast and slow conditioning, use per-position diffusion forcing, and apply a latency-adaptive schedule so each call emits fresh actions with one denoising step.

For GR00T-N1.7, that changes a strong but slow VLA into a manipulation policy that can run near 25 Hz on a real robot. The gains are clearest on dynamic, contact-rich tasks such as Catch Book, Don't Spill, Tidy Up Book, and Insert Box. This is not the final solution to humanoid manipulation: the paper notes that reactive data is hard to collect, proprioception conditioning is still limited, and current replanning is mostly local. But for real robot builders, πR² gives a valuable principle: semantic reasoning can be slow, but control reflexes must stay fast.

Related Posts

• NVIDIA GR00T + SONIC Whole-Body VLA
• π0-FAST and VLA training in LeRobot
• WholeBodyVLA: teleop, train, deploy

References

• πR² project page
• πR² paper on arXiv
• Official pi-r2-flow repository
• NVIDIA Isaac-GR00T
• NVIDIA GR00T-N1.7 on Hugging Face