SLA: Continual Pre-training π0.5 with 3D Spatial Supervision (+24%)

Reliable robotic manipulation requires not only semantic understanding but precise action execution in 3D space. The core problem: most modern VLA models — including π0.5 — inherit visual backbones from 2D-pretrained VLMs, with no explicit 3D spatial supervision baked in.

SLA (Spatial-Language-Action Model), from researchers at HKUST (Guangzhou), OLA Dimensions, Beijing Academy of AI, and Tsinghua University, proposes an elegant solution: rather than requiring explicit depth inputs or modifying the inference architecture, SLA implicitly aligns intermediate visual embeddings with 3D features from a frozen perception model — during continual pre-training, not just at fine-tuning time.

Result: +24.0 percentage points average improvement over π0.5 across 8 real-world deployment conditions.

Original paper: SLA: Scaling Spatial-Language-Action Model via Continual Pre-training — Fuhao Li et al., arXiv:2611.08373, Nov 05, 2026.

The Core Problem: 2D VLA Cannot "See" 3D Space

Imagine trying to grasp a cup placed behind a small obstacle. From an RGB camera, two objects at similar depth can appear overlapping. Yet the robot arm must navigate in 3D — determining exact relative positions, avoiding collisions, selecting the right grasp angle.

Current VLA models like π0.5 and OpenVLA inherit visual backbones from VLMs pretrained on 2D image-text pairs. Any 3D spatial understanding they develop — if any — emerges implicitly from demonstration data, without explicit supervision.

Existing approaches to add 3D understanding have tradeoffs:

• Depth cameras: add sensor modality, but depth sensors are noisy, bulky, not universal
• Point clouds: require dedicated encoder, adds architecture complexity
• 3D post-training only: builds spatial understanding too late, can't become a transferable foundation capability

SLA takes a different path: implicit spatial alignment during continual pre-training, with zero depth sensor requirements at inference.

Core Idea: A Frozen "Teacher" That Sees Space

The central intuition behind SLA: actions are generated from visual and language representations — so if visual tokens contain richer 3D information, actions will be more precise.

Instead of making visual tokens learn 3D from scratch, SLA uses a pre-trained 3D perception model (OmniVGGT) as a frozen "teacher" and aligns the VLA's intermediate visual embeddings with the teacher's spatial features. The teacher is never used at inference — only during training.

The pipeline:

1. Base model: Initialize from π0.5 pretrained checkpoint — no architecture changes
2. Frozen teacher: OmniVGGT receives RGB + optionally depth/calibration → extracts spatial features
3. Alignment projector: Learns to map VLA visual embeddings into teacher's feature space
4. Dual training objective:
   • Action loss (same as π0.5 — flow matching)
   • Alignment loss (cosine similarity between projected VLA embeddings and teacher features)
   • Total: L = L_action + α × L_align

The key insight: after training, discard the teacher, alignment projector, and layer-selection gates entirely. Inference is identical to π0.5.

Technical Architecture: 4 Key Components

1. External Spatial Supervision — OmniVGGT as Teacher

OmniVGGT is a pre-trained 3D perception foundation model that accepts:

• RGB images (required)
• Depth maps (optional — for datasets that have them)
• Camera intrinsics/extrinsics (optional — for calibrated rigs)

For datasets without depth, OmniVGGT still runs on RGB alone. Adding calibration can boost results further (e.g., from 42.8% to 46.0% in ablations).

Spatial features from OmniVGGT serve as "ground truth" about 3D structure to guide the VLA's learning.

2. Adaptive Layer Selection — Right Layer Per Domain

A key finding from the authors: the "best" intermediate VLA layer for alignment is not the same across different data domains. Desktop manipulation data differs from mobile arm data; a 7-DOF robot differs from a bimanual setup.

Feature-dependent gates solve this: instead of picking one fixed layer, the gates learn to adaptively combine visual embeddings from multiple VLA layers, with weights that depend on the input features themselves.

Ablations show adaptive selection consistently outperforms any fixed single-layer choice.

3. State Dropout — Preventing Robot-State Shortcuts

In robot manipulation, models can "cheat": if proprioceptive state (joint positions, end-effector pose...) is sufficiently informative, the model may predict the next action primarily from state while ignoring visual input.

Probabilistic state dropout randomly masks robot state during training, forcing the model to rely on visual cues. This not only improves generalization but makes spatial alignment meaningful — the model genuinely needs visual tokens to reason about action.

4. Rotation Representation

Instead of Euler angles or quaternions, SLA uses rotation vectors (axis-angle) for the action space. Ablations show this improves learning stability and end-effector precision.

Continual Pre-training Data

To scale spatial alignment, SLA needs diverse data. The authors unified 6 large public robot datasets:

Total: approximately 1.7 million episodes, ~14,330 hours of demonstrations spanning 1,504 tasks.

Unification required:

• State and action normalization: consistent end-effector representation
• Coordinate conventions: unified coordinate frame handling
• Downsampling: remove redundant frames, keep actionable transitions
• Filtering: remove low-quality demonstrations

Experimental Results

Simulation Benchmarks

Noteworthy on RoboCasa-Panda: SLA achieves 68.4% with just 50 demonstrations per task, while competing methods require 300–3,000 demos. This is strong evidence of data efficiency when spatial foundations are built early.

Real-World: +24.0 Percentage Points

Across 8 real-world deployment conditions covering:

• Bimanual coordination — two-arm coordination tasks
• Spatial generalization — object positions change at test time
• Dynamic generalization — environment changes during execution
• Few-shot cross-embodiment — adapt to a new robot with few demos
• Instruction following — diverse natural language commands

SLA averages 83.8% vs. 59.8% for π0.5. Crucially: SLA wins in all 8 conditions — not just on average.

Key Ablation: Pre-training vs. Post-training Alignment

The most important ablation compares three alignment strategies:

1. Only during post-training (like Spatial Forcing)
2. Only during continual pre-training
3. Both stages (full SLA)

Result: applying alignment in both stages wins at every tested data proportion, confirming that building 3D foundations early in pre-training is essential — not just a nice-to-have.

How to Run SLA — Practical Guide

Code is available via the project page at embodiedsla.github.io. The authors link to the OpenHelix-Team repository as the base.

Environment Setup

# Python 3.10+, CUDA 12.x
# GPU: A100/H100 for training; inference runs on RTX 3090+
conda create -n sla python=3.10
conda activate sla

git clone https://github.com/OpenHelix-Team/SpatialForcing
cd SpatialForcing
pip install -r requirements.txt

Inference Architecture — Zero Changes

The best part about SLA from a deployment perspective: nothing changes at inference. If you already have a π0.5 inference pipeline, SLA runs with the same code:

# Inference: exactly like pi0.5
# No OmniVGGT, no alignment projector, no layer gates needed
# Just use the SLA-trained checkpoint

from pi05_policy import Pi05Policy

policy = Pi05Policy.from_pretrained("path/to/sla-checkpoint")

# Identical to pi0.5 inference
action = policy.predict(
    image=rgb_obs,          # RGB only - no depth needed!
    language="pick up the cup",
    robot_state=joint_angles
)

Training Pipeline

For training SLA (or running your own continual pre-training):

# Step 1: Load pi0.5 checkpoint
base_policy = load_pi05_checkpoint("pi05-pretrained.pt")

# Step 2: Setup teacher (training only)
teacher = OmniVGGT.from_pretrained("omni-vggt-weights")
teacher.eval()  # Frozen — no gradient updates

# Step 3: Alignment projector + adaptive gates
alignment_projector = AlignmentMLP(
    vla_dim=1024,  # pi0.5 hidden dim
    teacher_dim=teacher.output_dim
)
layer_gates = AdaptiveLayerGates(num_layers=base_policy.num_layers)

# Step 4: Combined loss
def compute_sla_loss(batch, policy, teacher, projector, gates):
    # Action loss (flow matching like pi0.5)
    action_loss = policy.flow_matching_loss(batch)
    
    # Spatial alignment loss
    with torch.no_grad():
        spatial_features = teacher(
            rgb=batch["images"],
            depth=batch.get("depth"),          # Optional
            calibration=batch.get("calib")     # Optional
        )
    
    # Get intermediate visual embeddings from multiple layers
    vla_embeddings = policy.get_intermediate_embeddings(batch["images"])
    
    # Adaptive layer selection (weighted combination)
    aggregated = layer_gates(vla_embeddings)
    
    # Project and compute cosine similarity
    projected = projector(aggregated)
    align_loss = 1 - F.cosine_similarity(projected, spatial_features, dim=-1).mean()
    
    alpha = 0.1  # Alignment loss weight
    return action_loss + alpha * align_loss

# Step 5: Probabilistic state dropout
def apply_state_dropout(batch, dropout_prob=0.3):
    if random.random() < dropout_prob:
        batch["robot_state"] = torch.zeros_like(batch["robot_state"])
    return batch

Practical Tips for Custom Tasks

1. Is continual pre-training necessary?

If you only have a small task-specific dataset, applying the SLA alignment objective during post-training alone still provides benefits (this is essentially Spatial Forcing). For the full benefit documented in the paper, pre-training on a diverse multi-embodiment corpus is needed.

2. Rotation representation

Convert your action space to rotation vectors instead of Euler angles. Small change, meaningful improvement per ablations:

from scipy.spatial.transform import Rotation

def euler_to_rotvec(euler_angles):
    return Rotation.from_euler('xyz', euler_angles).as_rotvec()

3. Camera calibration

If your dataset includes camera intrinsics/extrinsics, provide them to OmniVGGT. If not, RGB-only still works (OmniVGGT can estimate depth from monocular cues).

4. State dropout schedule

Start with dropout_prob=0.3, gradually reduce during the final post-training phase so the model learns to balance visual and state cues rather than fully ignoring one.

SLA vs. Spatial Forcing: What Changed

SLA is a direct continuation of Spatial Forcing from the same OpenHelix-Team group. Key differences:

SLA doesn't "replace" Spatial Forcing — it's Spatial Forcing elevated to continual pre-training with additional technical improvements.

Strengths and Limitations

Key strengths:

• No inference architecture changes — fully compatible with existing π0.5 pipelines
• No depth sensor required at inference — RGB only
• Strong data efficiency: 50 demos instead of 3,000
• Wins across all 8 real-world conditions tested, not just on average
• Continual pre-training is scalable — more data, better foundation

Limitations to consider:

• OmniVGGT's full weights may not be open at time of writing — check availability
• Continual pre-training cost is high (1.7M episodes requires a GPU cluster)
• Official code documentation incomplete at paper release time
• Performance on non-Franka/UR robot arms not fully characterized

Broader Significance: Foundation Models Need 3D From Day One

SLA raises an important question for the community: should 3D understanding be built into foundation models during pre-training, or just added as post-training fine-tuning?

SLA's ablations clearly favor the first approach — analogous to how LLMs benefit from being pre-trained on code and math before task-specific fine-tuning. VLA models benefit from having spatial understanding baked into the foundation, not patched in afterward.

This points toward a next generation of VLA systems that don't just "learn better from demos" but "understand space more deeply from the ground up" — a distinction that likely matters more as tasks become more complex and environments less structured.

Related Posts

• SpatialVLA and 3D Perception for Robot Manipulation
• OmniManip: Spatial Reasoning for Manipulation
• STAR: Improving π0.5 on LIBERO-PRO Without Retraining