Robo-ValueRL: Deploy VLA-RL Precision Manipulation for Humanoids

You have probably seen robot manipulation demos that look impressive on YouTube — until the real task lands in front of you: inserting a semiconductor chip or disassembling precision electronics where millimeter-level accuracy is non-negotiable. Standard VLA models start falling apart. They train indiscriminately on mixed-quality data, lose track when the gripper occludes the object, and have no self-assessment mechanism to stabilize online learning.

Robo-ValueRL — an open-source framework released in July 2026 by X-Humanoid (Beijing Innovation Center of Humanoid Robotics) in collaboration with GeWu-Lab at Renmin University of China — was built specifically to fix these three failure modes. This guide walks you from "why this framework exists" through "up and running on real hardware" in one sitting.

The Problem: Why Standard VLA Falls Short

Think of a VLA model as a junior engineer who learned entirely from watching senior colleagues (imitation learning). Capable, but with three structural blind spots:

Problem 1: Dirty training data. Across 240 hours of collected demonstrations, quality varies enormously. Some teleop sessions have jitter, mid-task hesitations, or outright failures that still got recorded. Training without discrimination means the model learns bad behaviors alongside good ones.

Problem 2: No self-assessment. During manipulation, the gripper frequently occludes the target object. A single-frame observation model cannot reason "what did I do in the last 16 steps?" to infer current state, leading to poor decisions under low-visibility conditions.

Problem 3: Unstable online adaptation. When fine-tuning on the real robot, data distribution shifts continuously. Without a guiding mechanism, the policy oscillates rather than improves — the classic instability problem in offline-to-online RL.

Robo-ValueRL addresses all three with one central mechanism: history-conditioned value estimation — a value function conditioned on the sequence of past observations and actions.

What Is Robo-ValueRL?

Simply put: Robo-ValueRL adds a "quality judge" to the VLA pipeline. This judge (the value estimator) reviews the full history of the robot's actions and scores them: "How good was that action? How far along is the task?"

This score drives three downstream benefits:

1. Offline data filtering: only high-quality demonstrations inform training
2. Quality-conditioned pretraining: policy learns to prefer good actions
3. Stable online adaptation: the value function acts as an anchor preventing policy drift

The framework builds on OpenPI (π₀/π₀.₅) as the VLA backbone and LeRobot v2.1, so you are not starting from scratch.

Benchmark results:

• Chip insertion (±0.5mm tolerance): 86% success rate
• Block disassembly: 84% success rate
• Trained on 240 hours offline demos + ~3,000 online rollouts

The 4-Stage Architecture

Stage 1: Value Estimation — The History-Aware Judge

The value estimator receives sequences of multi-view camera observations and past actions (history-conditioned), then predicts remaining time — how many steps until task completion.

This distinguishes it from standard value functions that only observe the current state. By conditioning on history, the estimator can reason through occlusion and reduce uncertainty from partial observations.

# Value estimator structure (conceptual)
class HistoryConditionedValueEstimator(nn.Module):
    def __init__(self, obs_horizon=16, n_cameras=3):
        super().__init__()
        self.visual_encoder = VisionTransformer(...)     # 3-camera input
        self.history_encoder = TransformerEncoder(seq_len=obs_horizon)
        self.value_head = nn.Linear(hidden_dim, 1)       # remaining time
    
    def forward(self, obs_history, action_history):
        visual_feats = self.visual_encoder(obs_history)
        history_feats = self.history_encoder(action_history)
        fused = torch.cat([visual_feats, history_feats], dim=-1)
        return self.value_head(fused.mean(dim=1))

Remaining time predictions are converted to task progress (0.0 = just started, 1.0 = complete). The delta between consecutive timesteps is the value difference — a measure of how productive that action was.

Stage 2: Quality Annotation — From Scores to Labels

Once the value estimator is trained on a held-out split, the full offline dataset is annotated with quality labels derived from value differences:

This transforms the raw teleop dataset into a quality-annotated dataset without any manual re-labeling.

Stage 3: Offline Pretraining — Learning with Selectivity

The policy is trained as a quality-conditioned consistency policy — a diffusion-style policy conditioned on quality labels. At inference time, always condition on GOOD to steer the policy toward optimal actions.

# Offline pretraining
cd robo_valuerl
bash scripts/train_robo_valuerl_offline.sh
# Checkpoint saved to: outputs/offline_pretraining/

Consistency policy has a practical advantage over full diffusion: single-step inference instead of iterative denoising, meeting real-time control requirements.

Stage 4: Online Adaptation — Fine-Tuning on the Real Robot

This is the stage where most VLA-RL pipelines break. Robo-ValueRL uses a Residual Adaptation Module combined with RTC (Real-Time Chunking) for stability.

The residual adapter is a small module added on top of the pretrained policy. Only the adapter is updated during online steps — this avoids catastrophic forgetting while enabling targeted correction. The value function from Stage 1 continues evaluating online rollouts, prioritizing high-value samples in the replay buffer.

# Online adaptation
bash scripts/train_robo_valuerl_online.sh
# Requires active connection to robot server via socket

Installation

Hardware Requirements

Tip: If you don't have 8× A100, reduce GPU count and increase gradient_accumulation_steps proportionally. Community reports successful training with 4× A100 using gradient_accumulation_steps=2.

Clone and Setup

# Clone repo
git clone https://github.com/Open-X-Humanoid/Robo-ValueRL.git
cd Robo-ValueRL

# Single-command install (creates conda env + installs all deps)
bash install_simple.sh

install_simple.sh automatically:

1. Creates robo_valuerl conda environment
2. Installs LeRobot v2.1
3. Installs OpenPI (π₀ backbone)
4. Applies necessary PyTorch patches to transformers

# Activate
conda activate robo_valuerl

# Verify
python -c "import robo_valuerl; print('OK')"
python -c "import openpi; print('OK')"

Repository Layout

Robo-ValueRL/
├── robo_valuerl/
│   ├── value_estimator/      # Stage 1: History-conditioned value model
│   ├── quality_annotation/   # Stage 2: Rank-based labeling
│   ├── offline_pretraining/  # Stage 3: Consistency policy
│   ├── online_adaptation/    # Stage 4: Residual adapter + RTC
│   └── utils/
├── scripts/
│   ├── train_robo_valuerl_value_estimator.sh
│   ├── train_robo_valuerl_offline.sh
│   └── train_robo_valuerl_online.sh
├── configs/
│   └── default.yaml
└── install_simple.sh

Training Pipeline Step-by-Step

Step 1: Prepare Your Dataset

The framework expects LeRobot v2.1 format. Convert raw teleop recordings (video + joint states) with:

python robo_valuerl/utils/convert_to_lerobot.py \
    --input_dir /path/to/raw_demos \
    --output_dir /path/to/lerobot_dataset \
    --robot_type tienkung_dual_arm \
    --n_cameras 3

X-Humanoid's official dataset is available on HuggingFace:

huggingface-cli download X-Humanoid/Robo-ValueRL \
    --repo-type dataset \
    --local-dir ./data

Step 2: Train the Value Estimator

cd robo_valuerl
bash scripts/train_robo_valuerl_value_estimator.sh \
    --dataset_path ./data/lerobot_dataset \
    --obs_horizon 16 \
    --n_cameras 3 \
    --epochs 50 \
    --batch_size 64

Training takes ~6-8 hours on a single A100. Best checkpoint is selected by validation loss on remaining-time prediction.

Monitoring tip: Watch value_calibration_error in logs. If it stays above 0.15 after 20 epochs, increase obs_horizon or add more demonstration data.

Step 3: Annotate Quality Labels

python robo_valuerl/quality_annotation/annotate_dataset.py \
    --dataset_path ./data/lerobot_dataset \
    --value_ckpt ./outputs/value_estimator/best.ckpt \
    --output_path ./data/annotated_dataset \
    --good_threshold 0.33 \
    --bad_threshold 0.33

This pass adds a quality_label column to every timestep in the dataset. Runtime: ~2-3 hours for 240h of data.

Step 4: Offline Pretraining

# Distributed across 8 GPUs
torchrun --nproc_per_node=8 \
    robo_valuerl/offline_pretraining/train.py \
    --dataset_path ./data/annotated_dataset \
    --model_name openpi/pi0.5-base \
    --batch_size 256 \
    --lr 1e-4 \
    --epochs 100 \
    --output_dir ./outputs/offline_pretraining

The consistency policy is initialized from pretrained π₀.₅ weights and fine-tuned on the annotated dataset. Quality conditioning is implemented via a prefix token ([GOOD], [MEDIUM], [BAD]) prepended to the input sequence. At inference, always use [GOOD].

Step 5: Online Adaptation

Online adaptation requires a live robot connection. The architecture uses a policy server / robot client pattern over sockets:

# Terminal 1: Start policy server on GPU machine
python robo_valuerl/online_adaptation/policy_server.py \
    --checkpoint ./outputs/offline_pretraining/best.ckpt \
    --port 8765

# Terminal 2: Start robot client on controller
python robo_valuerl/online_adaptation/robot_client.py \
    --server_ip 192.168.1.100 \
    --port 8765 \
    --robot_config configs/tienkung_dual_arm.yaml

Online adaptation runs automatically: the robot executes the policy, the value estimator scores outcomes, high-value rollouts enter the replay buffer, and the residual adapter updates every N rollouts.

Safety note: max_correction_magnitude in configs/default.yaml caps how much the residual adapter can deviate from the pretrained policy. Start at 0.1 and increase gradually after verifying robot behavior at each step.

Experimental Results

Two benchmark tasks:

Chip Insertion — representative precision manufacturing task:

• Tolerance: ±0.5mm
• Robo-ValueRL: 86% success rate
• Behavior cloning baseline: ~61%
• Policy autonomously corrects failed grasps using value-guided trajectory adjustment

Block Disassembly — generalization benchmark:

• Longer action sequences required
• Robo-ValueRL: 84% success rate
• Generalizes to object orientation variations not present in training data

Key ablation: Removing history conditioning (falling back to single-frame observation) drops chip insertion success rate to 71% — demonstrating that the historical observation window is the primary driver of performance, especially under occlusion.

Compared to approaches like SimpleVLA-RL and ProcVLM dense reward: Robo-ValueRL focuses on value reliability rather than reward density, yielding more stable online adaptation in settings where a ground-truth reward function is unavailable.

Robot Configuration: X-Humanoid TienKung

The framework is designed for the TienKung dual-arm humanoid but the socket-based policy server/client architecture can be adapted to other platforms by:

1. Writing a custom RobotClient implementing BaseRobotClient
2. Updating robot_config.yaml with joint limits and action space
3. Remapping camera configuration (framework defaults to 3 cameras: wrist-left, wrist-right, overhead)

# configs/tienkung_dual_arm.yaml
robot:
  n_arms: 2
  dof_per_arm: 7
  cameras:
    - name: wrist_left
      resolution: [640, 480]
      fps: 30
    - name: wrist_right
      resolution: [640, 480]
      fps: 30
    - name: overhead
      resolution: [1280, 720]
      fps: 30
  action_space: joint_positions
  control_freq: 50  # Hz

Real-World Applications

X-Humanoid targets three industry verticals with this framework:

1. Semiconductor assembly — IC chip insertion, micro-soldering, PCB inspection
2. Precision electronics — connector mating, ribbon cable routing, camera module assembly
3. Medical devices — sterile assembly of instruments requiring sub-millimeter accuracy

Key advantage over traditional industrial robot arms (ABB, FANUC): Robo-ValueRL does not require rigid jig fixtures. The policy learns to adapt to positional variation in object placement, reducing setup time and fixture cost.

Resources

• GitHub: Open-X-Humanoid/Robo-ValueRL
• Project Page: gewu-lab.github.io/Robo-ValueRL
• Paper (arXiv): arXiv:2607.09866 — Robo-ValueRL: Reliable Value Estimation for Offline-to-Online RL
• HuggingFace Models: X-Humanoid/Robo-ValueRL
• HuggingFace Dataset: X-Humanoid/Robo-ValueRL (dataset)
• X-Humanoid: x-humanoid.com

Conclusion

Robo-ValueRL represents a pragmatic and immediately deployable direction: rather than designing complex reward functions or collecting millions of rollouts, the framework leverages value estimation from existing teleop data to bootstrap a policy capable of precision manufacturing.

Three things to take away:

1. History conditioning solves occlusion and partial observability without SLAM or explicit state estimation
2. Quality annotation is fully automatic from value differences — no manual re-annotation needed
3. Residual adapter makes online adaptation lightweight and stable — no need to retrain the backbone

If you are working on high-precision manipulation tasks and already have a teleop demonstration dataset, Robo-ValueRL is a serious starting point worth your time.

Related Posts

• SimpleVLA-RL: A Simple VLA-RL Framework from Zero to Deployment
• HiL-SERL: Real-Time Online RL for Robots with Human-in-the-Loop
• ProcVLM: Automatic Dense Rewards for VLA-RL Without Manual Annotation