GroundingPI: XPENG's 4B Grounding Model Beats GPT-6 Astra

There is a quiet paradox at the heart of modern VLA (Vision-Language-Action) research: action policies grow more sophisticated each month, yet the "eyes" feeding them — the vision backbones — are borrowed from general-purpose models never optimized for the one job that matters most in manipulation: seeing exactly which object to touch and precisely where.

XPENG, collaborating with researchers from Peking University and the University of Hong Kong, proposes a different path. Instead of patching a general backbone onto a robot pipeline, build a dedicated grounding model from the ground up — one whose sole purpose is accurate perception. The result is GroundingPI: a 4-billion-parameter model that averages 73.68% across 34 grounding benchmarks (above GPT-6 Astra at 71.54%), and demonstrably improves downstream VLA manipulation with open weights and code.

The Core Problem: VLA Models Are Perception-Blind

Imagine asking someone to pick up a ballpoint pen, but they can only perceive "elongated cylindrical object, blue" with no ability to localize it precisely when surrounded by clutter. That, in essence, is what current VLA models experience at inference time.

Today's dominant VLA backbones (general VLMs, video-generation models) excel at broad scene understanding but struggle with:

• Small and occluded objects: A 5mm bolt on a circuit board, a component half-hidden under another part
• Dense scenes: Ten objects on a shelf — pick exactly one
• Coordinate precision: The grasp point must be within centimeters, not "somewhere in that region"
• Low latency: Closed-loop control cannot afford a slow vision pipeline

GroundingPI is designed to close every one of these gaps, with an architecture and training pipeline built exclusively around precise localization.

What Is GroundingPI?

GroundingPI is a 4-billion-parameter grounding foundation model targeting Physical Intelligence — the class of systems (robots, autonomous vehicles) that must interact with the physical world rather than produce text.

Its defining design choice: represent points and bounding boxes as quantized coordinates in a shared vocabulary. Every perception task — open-vocabulary detection, referring grounding, OCR, GUI grounding, layout detection, visual prompting — becomes a single problem: "generate a coordinate sequence conditioned on a language query." No separate detection head, no task-specific post-processing. One autoregressive model, one vocabulary, eleven perceptual capabilities.

Architecture Deep Dive

Visual Encoder: MoonViT-V2 (Kimi K3)

The visual encoder is MoonViT-V2, introduced in Kimi K3. It processes high-resolution images with the detail fidelity needed to localize small and densely packed objects — a property that generic VLM encoders frequently sacrifice for semantic breadth.

Language Decoder: Qwen3-4B

The language component is Qwen3-4B, an autoregressive decoder where every coordinate is generated as a standard next-token prediction. This unification means spatial reasoning and language reasoning share the same representational space — coordinates are just another kind of "word" the model learns to produce.

Projector

A two-layer MLP with GELU activation and output normalization aggregates adjacent 2×2 visual features and projects them into the language embedding space. The design is deliberately simple: the heavy lifting happens in the two backbone components.

Coordinate Tokenization

The key insight: a bounding box [x1, y1, x2, y2] is encoded as four integer tokens drawn from Qwen3's vocabulary. The model learns, end-to-end, to associate language descriptions with spatial positions. No anchor boxes, no regression heads, no non-maximum suppression.

Three-Stage Training Pipeline

GroundingPI's training is structured in three distinct stages, each targeting a different learning objective:

Stage 1: Multimodal and Spatial Pretraining

The foundation stage. The model is trained on broad multimodal data: image-text pairs, detection annotations, referring expressions, OCR data, GUI screenshots, document layouts. Beyond public datasets, the authors build "dedicated data engines" — automated pipelines that generate high-quality grounding annotations at scale. This stage instills the model's basic perceptual vocabulary.

Stage 2: Supervised Fine-Tuning (SFT)

Token-level cross-entropy loss teaches the model to produce structurally valid outputs: the precise syntax of a bounding box sequence, the format of a pointing coordinate, the layout of an OCR result. SFT converts pretraining knowledge into well-formed grounding outputs.

Stage 3: GRPO Reinforcement Learning

The most distinctive stage. Group Relative Policy Optimization (GRPO) samples G=8 responses per image–query pair and computes rewards to update the model. Two complementary reward signals:

R_set (Coverage): For each ground-truth box, find the prediction with the highest IoU and validate its class label. This catches missing objects.

R_strict (Quality): Multi-threshold F1, localization accuracy, format correctness, count, ordering. Penalizes duplicates and oversized boxes.

Combined advantage: Ã_i = 0.7 × Z(R_set,i) + 0.3 × Z(R_strict,i)

where Z() standardizes rewards within a prompt, and advantages are further standardized across the response batch. Critically: only language parameters are updated; the SFT checkpoint serves as a frozen reference. This prevents catastrophic forgetting and stabilizes training under reward-based updates.

For OCR, Hungarian matching evaluates both text string and geometry — exact text matters here, not just IoU overlap.

Benchmark Results: 34 Grounding Tasks

GroundingPI is evaluated against 44 baselines across 34 benchmarks spanning 11 perceptual capabilities:

Result: 73.68% average, versus GPT-6 Astra at 71.54% with thinking effort set to High. GroundingPI achieves this with significantly fewer parameters.

Where GroundingPI leads: Dense200, VisDrone, referring grounding, spatial grounding, dense OCR. All the domains where precise localization in complex scenes is paramount.

Where Astra still leads: ScreenSpot-Pro (high-precision GUI grounding). One notable limit to keep in mind.

All evaluations are averaged over ten runs (five seeds, two runs per seed) to reduce variance — a methodological detail worth noting given how narrow the margin over Astra is.

Robot Manipulation Results

This is what matters most for robotics practitioners.

Integration Architecture: π-style Flow Matching + Action DiT

GroundingPI serves as a frozen visual backbone feeding into an Action DiT (Diffusion Transformer for Actions), following the layer-wise connection pattern introduced in StarVLA:

Observation image + language instruction
        ↓
GroundingPI (frozen) encodes once
        ↓
Intermediate features projected and resampled
        ↓
Injected into Action DiT via cross-attention (layer-wise)
        ↓
Action DiT denoises noisy action chunk → continuous robot controls

The foundation model is frozen throughout manipulation training. Only the Action DiT is updated. This means GroundingPI can be dropped into any π-style VLA pipeline as a plug-in backbone.

RoboTwin 2.0: Out-of-Distribution Generalization

RoboTwin 2.0 evaluates four out-of-distribution (OOD) settings designed to stress-test generalization. GroundingPI outperforms every mainstream backbone in all four OOD settings, with improvements up to 24.8% relative to the strongest baseline.

The comparison covers four backbone families under identical Action DiT, robot data, and training budget — the only variable is the pretrained foundation. This controlled setup makes the result more credible than a full system comparison where many factors vary simultaneously.

RoboCasa-GR1: Data Efficiency

The headline result: GroundingPI trained on 50% of demonstrations outperforms all baselines trained on 75%. A 25% reduction in required demonstration data for equal or better performance.

Robot data is expensive. Teleoperation demonstrations take human time, robot time, and careful quality control. A backbone that learns faster from less data has significant practical implications for deploying manipulation policies in new environments.

Autonomous Driving: nuScenes

GroundingPI was also tested as a visual backbone for autonomous driving on nuScenes, achieving an average open-loop L2 error of 0.296 m. This confirms that strong perceptual grounding transfers across domains — the same model that improves robot arm manipulation also improves trajectory planning for autonomous vehicles.

Installation and Usage

Setup

# Clone the repository
git clone https://github.com/groundingpi/GroundingPI.git
cd GroundingPI

# Create environment (conda recommended)
conda create -n groundingpi python=3.10 -y
conda activate groundingpi

# Install dependencies
pip install -e .

Download Model Weights

from huggingface_hub import snapshot_download

# Download from HuggingFace Hub
snapshot_download(
    repo_id="GroundingPI/GroundingPI",
    local_dir="./checkpoints/groundingpi-4b"
)

Basic Inference: Object Grounding

import torch
from PIL import Image
from groundingpi import GroundingPIModel, GroundingPIProcessor

# Load model and processor
model = GroundingPIModel.from_pretrained("./checkpoints/groundingpi-4b")
processor = GroundingPIProcessor.from_pretrained("./checkpoints/groundingpi-4b")
model = model.cuda().eval()

# Load image
image = Image.open("robot_workspace.jpg")

# Natural language query
query = "Find the red cup on the table"

# Preprocess
inputs = processor(images=image, text=query, return_tensors="pt").to("cuda")

# Run inference
with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=100,
        do_sample=False
    )

# Decode spatial output
result = processor.decode_grounding(outputs[0])
print(f"Boxes: {result['boxes']}")   # [[x1, y1, x2, y2], ...]
print(f"Points: {result['points']}") # [[x, y], ...]

Using GroundingPI as a VLA Backbone

from groundingpi import GroundingPIForVLA

# Load backbone — freeze for downstream training
backbone = GroundingPIForVLA.from_pretrained(
    "./checkpoints/groundingpi-4b",
    freeze_backbone=True  # Only train the Action DiT downstream
)

def encode_observation(image, instruction):
    """Extract layer-wise features for Action DiT injection."""
    with torch.no_grad():
        features = backbone.extract_features(
            images=image,
            text=instruction,
            return_intermediate=True  # Returns dict of layer features
        )
    return features

# Inside Action DiT training loop
obs_features = encode_observation(obs_image, task_description)
predicted_action = action_dit(
    noisy_actions=noisy_action_chunk,
    visual_features=obs_features,  # Injected via cross-attention
    robot_state=proprioception
)

Supported Query Types

The shared vocabulary covers all 11 perceptual capabilities:

# Open-vocabulary detection
query = "Detect all objects in the image"

# Referring grounding (pick one specific object)
query = "Find the small blue cube next to the yellow cylinder"

# Spatial relationship grounding
query = "What is to the left of the red cup?"

# Grasp point prediction
query = "Point to the optimal grasp location on the mug handle"

# GUI / touchscreen grounding
query = "Locate the 'Confirm' button on the interface"

# Scene-aware OCR
query = "Read the text on the product label facing the camera"

Strengths and Limitations

Why This Architecture Works

Clear scaling law: Both manipulation and driving performance improve as pretraining data and model scale increase. This is the most important property for a foundation model — it means the approach has a predictable path to improvement.

Dense grounding is the key capability: Systematic ablation across the 11 perceptual capabilities shows that dense grounding (finding all objects in a cluttered scene) provides the largest downstream benefit for both manipulation and driving. If you only had one capability to include, dense grounding would be it.

OCR as a catalyst: The surprising finding is that OCR training improves other perceptual capabilities beyond text reading. The hypothesis is that combining exact text matching with precise geometry teaches the model something fundamental about spatial consistency.

Data efficiency compounds: Better perception means faster policy learning. This compounding effect makes GroundingPI particularly valuable in settings where demonstration collection is the bottleneck.

Honest Limitations

ScreenSpot-Pro: GPT-6 Astra retains a meaningful advantage on high-precision GUI grounding. GroundingPI is not the best choice for screen interaction tasks.

Prototype-stage maturity: XPENG explicitly states that all results are based on simulation and offline evaluation. This does not establish readiness for commercial real-world deployment.

No independent replication yet: All benchmark numbers come from the authors' own evaluation. The robotics community has not yet independently reproduced these results.

Narrow margin over Astra: 73.68% vs. 71.54% is a real improvement, but not a large one. The more compelling argument is the manipulation-specific performance gap, not the aggregate grounding number.

What to Build with GroundingPI

For robotics researchers and engineers:

1. Replace your VLA vision backbone: If you are running a π-style flow-matching policy, GroundingPI is a drop-in replacement for your current backbone. Benchmark on your task before committing — but the RoboTwin OOD results suggest it generalizes better.

2. Data-limited manipulation problems: If you have fewer than 500 demonstrations for a new manipulation skill, GroundingPI's data efficiency advantage may be decisive.

3. Dense manipulation tasks: Bin picking, kitting, sorting tasks where many similar objects appear together are exactly the domain where dense grounding helps most.

4. Perception-only module: Even outside VLA, GroundingPI works as a standalone spatial understanding module for scene interpretation, task planner grounding, or failure detection.

Conclusion

GroundingPI asks a sharp question: how much of VLA manipulation performance is bottlenecked not by the action policy, but by perception quality? The empirical answer from RoboTwin 2.0 (up to 24.8% OOD improvement) and RoboCasa-GR1 (50% data matches 75%) suggests the answer is "a significant fraction."

The architectural bet — dedicated grounding pretraining over general-purpose VLM backbones — is not new as a concept, but GroundingPI is the first open model to demonstrate it convincingly at manipulation scale. With weights on HuggingFace and code on GitHub, the community now has a concrete starting point to test whether this advantage holds on their own tasks.

Paper: GroundingPI on arXiv
Code: github.com/groundingpi/GroundingPI
Weights: huggingface.co/GroundingPI/GroundingPI

Related Posts

• StarVLA: Modular VLA with Layer-wise Connection for Robot Manipulation
• RoboTwin 2.0: Dual-Arm Manipulation Benchmark with OOD Generalization
• VLA Models: From OpenVLA to Pi0 — A Comprehensive Overview