BridgeVLA++: Training VLA for 3D Manipulation with Spatio-Temporal Memory

Here's a scenario you've probably seen before: a robot executes a manipulation sequence flawlessly on the first attempt, but the moment an object gets partially occluded by its own gripper — or the task requires remembering which box was already used — performance collapses entirely. This is the memory problem in VLA (Vision-Language-Action) models. No matter how capable the underlying language model, without explicit memory the policy has no way to track what it just did or where objects currently reside.

BridgeVLA++ (arXiv 2608.05042), from the Chinese Academy of Sciences and ByteDance Seed, tackles this head-on. Submitted to IEEE TPAMI in August 2026, it extends the original BridgeVLA (NeurIPS 2025) with a unified spatio-temporal memory module: temporal memory answers "what do I do next?", spatial memory answers "exactly where do I act?". The result: 93.7% on RLBench, 96.0% on RMBench (the dual-arm memory-dependent benchmark), state-of-the-art across 5 simulation benchmarks, and validated on 2 real-robot platforms.

What is BridgeVLA? The Foundation You Need First

BridgeVLA++ builds on BridgeVLA (NeurIPS 2025), so a quick recap is essential. BridgeVLA's core insight is elegant: instead of processing 3D point clouds in 3D space (hard for pretrained VLMs to understand), it renders point clouds into three orthographic 2D images (top, front, side views), then uses a VLM (PaliGemma 3B) to predict 2D heatmaps — each pixel representing the probability that "this is the next action location". The robot recovers the 3D action coordinates from these heatmaps.

This keeps input-output alignment with the VLM's 2D pretraining distribution, while still exploiting 3D geometry. Fine-tuning requires just 100 demonstrations per task in simulation, 10 for real-world.

Where BridgeVLA falls short: no memory. At each step, the policy sees only the current observation. Tasks requiring history — which container holds what? what step did I just complete? where is the object that's now behind my gripper? — cause BridgeVLA to fail. On RMBench, a 9-task benchmark specifically designed for memory-dependent dual-arm coordination, BridgeVLA achieves… 18.9%.

BridgeVLA++: How Spatio-Temporal Memory Works

BridgeVLA++ leaves the BridgeVLA core unchanged. Instead, it injects two memory modules into the VLM's patch-token space at different stages of the coarse-to-fine pipeline.

Temporal Memory (𝒯ₜ) — "What Do I Do Next?"

Temporal memory operates at the coarse stage, where the model determines the broad region and type of action. It stores three categories of information:

1. Initial anchor views (𝐀₀): Snapshots from the episode's first observation. This is the "ground truth" of the initial state — before the robot touched anything.

2. Neighboring keyframes (ℋₜⁿᵇʳ): The N=2 most recent executed keyframes. Tells the model "here's what I just did."

3. Sub-goal keyframes (ℋₜˢᵘᵇ): Frames selected adaptively by a learned module that marks task milestones — "lid opened", "object placed inside". Trained with binary cross-entropy loss to learn which frames are worth remembering.

All three are injected into the VLM via cross-attention: current tokens query memory tokens, asking history before committing to the next action.

Spatial Memory (𝒮ₜ) — "Exactly Where Do I Act?"

Spatial memory operates at the fine stage, where precise localization matters. The challenge: when the gripper holds an object, the gripper itself may occlude that object in the current observation. You can't localize what you can't see.

The solution: store the colored point cloud from the first observation (before the robot disturbed anything). When executing a fine-stage action, the model re-renders this reference point cloud using the same zoom and viewpoint as the current observation, producing a clean geometric reference. No robot arm appears in this reference image — the target object is clearly visible.

Computational Overhead

The entire memory module adds only 269.77M parameters to the 2.92B PaliGemma backbone — a 9.2% overhead. Inference latency increases from 0.35s to 0.57s per step on an RTX 4090. A very reasonable trade-off for the gains shown below.

Installation and Environment Setup

Clone the repository and install the benchmark-specific conda environment:

git clone https://github.com/BridgeVLA/BridgeVLA.git
cd BridgeVLA
# main branch = BridgeVLA++
# git checkout bridgevla  # for the original NeurIPS 2025 version

Each benchmark ships with its own idempotent installer:

# RLBench — the most common benchmark (18 tasks)
bash finetune/RLBench/install_rlbench.sh

# COLOSSEUM — generalization under 14 distribution shifts
bash finetune/RLBench/install_rlbench.sh   # install RLBench first
bash finetune/Colosseum/install_colosseum.sh

# GemBench / MemoryBench
bash finetune/GemBench/install_gembench.sh

# RMBench — dual-arm memory tasks
bash finetune/RMBench/install_rmbench.sh

# Pre-training (only if training from scratch)
bash pretrain/install_pretrain.sh

Download datasets for the benchmarks you need:

# All 5 benchmarks
bash scripts/download_datasets.sh rlbench colosseum gembench memorybench rmbench

# Individual benchmark with auto-extract
bash scripts/download_datasets.sh rlbench --extract
bash scripts/download_datasets.sh rmbench --extract

Note: RLBench requires pre-built keyframe caches. The download script includes them automatically.

Pre-training: Teaching the Model to See

BridgeVLA++ uses a two-stage training process. Stage 1 is grounding pre-training — teaching the PaliGemma backbone to predict 2D heatmaps from language instructions:

# Pre-training on 120K RoboPoint object-detection data
# Requires 8×A100, ~2 hours
bash pretrain/pretrain.sh

The pre-training dataset is the RoboPoint object-detection split (120K samples) — pairs of (image, text instruction, bounding box). The model learns: "when the instruction says 'pick up the red cup', focus the heatmap on the red cup's location." This alignment step is the most critical — after pre-training, the model has learned to attend to the correct objects.

If you prefer not to pre-train from scratch, download the pretrained checkpoint:

bash scripts/download_checkpoints.sh pretrained

Fine-tuning: Teaching the Policy to Manipulate

After pre-training, fine-tune on your target benchmark. BridgeVLA++ requires only 100 demonstrations per task in simulation — far less than most VLA approaches:

# Fine-tune on RLBench (18 manipulation tasks)
bash finetune/RLBench/train.sh

# Fine-tune on COLOSSEUM
bash finetune/Colosseum/train.sh

# Fine-tune on GemBench
bash finetune/GemBench/train.sh

# Fine-tune on MemoryBench (single-arm memory tasks)
bash finetune/memoryBench/train.sh

# Fine-tune on RMBench (dual-arm memory tasks)
bash finetune/RMBench_vla/train.sh

During fine-tuning, the model optimizes a dual loss:

L_total = L_est + λ_check × L_check

Where:

• L_est: action prediction loss (heatmap regression)
• L_check: binary cross-entropy for adaptive keyframe selection — teaches the module which frames deserve sub-goal memory slots
• λ_check = 0.1 by default

Data augmentation applies consistently across current observations, memory samples, and ground-truth actions. This is important: inconsistent augmentation would create mismatch between what memory stores and what the current observation looks like.

Evaluation: SOTA Across 5 Benchmarks

Running Evaluation

# Evaluate on RLBench
bash finetune/RLBench/eval.sh

# GemBench and MemoryBench use a server-client setup (2 terminals)
# Terminal 1:
bash finetune/GemBench/start_server.sh
# Terminal 2:
bash finetune/GemBench/eval.sh

Results Summary

The most striking number is RMBench: from 18.9% to 96.0% — a +77 percentage point jump. This is the clearest evidence that memory is not a nice-to-have but a prerequisite for memory-dependent tasks. Without memory, dual-arm coordination is nearly impossible (the policy has no record of what the other arm just did, or where objects were placed). With BridgeVLA++, the full episode history is available via temporal memory.

MemoryBench is equally impressive: 99.7% ± 0.3% — near-perfect. These 3 tasks specifically require tracking object identity and location across multiple steps.

Real-World Results

# Real-robot deployment (requires IP config and camera setup)
bash real_robot/install_train.sh        # on GPU server
bash real_robot/install_deployment.sh   # on robot workstation
bash real_robot/eval_franka.sh          # run evaluation

Real-world performance:

• Franka Research 3 (13 tasks): 96.9% success with 10 demonstrations per task
• Franka with only 3 demos/task: 95.4% — exceptionally data-efficient
• Dobot CR5A (memory tasks): 93.3% vs SAM2Act+'s 30.0%

Camera: static ZED 2i stereo camera. No wrist camera required — the point cloud provides sufficient 3D information.

Bimanual: When Two Arms Must Coordinate

One underappreciated feature of BridgeVLA++ is native bimanual support. By sharing spatial memory between both arms (same reference point cloud, same geometric space), the two arms can coordinate without a separate coordination module.

RMBench (9 dual-arm tasks) results:

• No memory: 18.9% (essentially random failure)
• BridgeVLA++: 96.0%

This demonstrates true cross-embodiment scalability: the same framework, the same training recipe, working well on both single-arm and dual-arm setups.

Comparison with Other VLA Approaches

BridgeVLA++ is uniquely strong on memory-dependent tasks — this is the largest gap relative to existing methods, and arguably the most important gap for real-world deployment.

When Should You Use BridgeVLA++?

BridgeVLA++ is the right choice when:

• Multi-step tasks require history tracking: pick → place → close lid. Memory keeps track of which step is complete.
• Data collection is expensive: 10 demos for real-world deployment is extremely low. If robot time or human teleop time is limited, this matters.
• Occlusion is unavoidable: when the gripper blocks the target object's view, spatial memory maintains accurate localization.
• Bimanual coordination is required: native dual-arm support via shared memory is a unique strength.

Consider alternatives when:

• The task is simple and memoryless: single-step pick-and-place without multi-step dependencies. The original BridgeVLA (bridgevla branch) is faster and sufficient.
• Latency is critical: 0.57s per step. If you need sub-100ms control loops, a different architecture is needed.

Conclusion

BridgeVLA++ demonstrates a principle that's often overlooked in VLA design: robots need to remember. Not everything — but the right things. Temporal memory selectively retains important keyframes; spatial memory preserves initial geometry to resolve occlusions. Together they let the robot know both "what to do next" and "exactly where to do it."

With just 9.2% parameter overhead and SOTA results across 5 benchmarks, BridgeVLA++ is one of the most production-ready VLAs for 3D manipulation tasks currently available. Code and checkpoints are public at github.com/BridgeVLA/BridgeVLA.

Related Posts

• ResVLA: Anchoring VLA Policy with Residual Diffusion Bridge (ICML 2026)
• FM-VLA: Force Memory Token for Contact-Rich Robot Manipulation
• Spatial VLA: From Text Instructions to 3D Action Space