DSPv2: Dense Policy for Whole-Body Mobile Manipulation

The Whole-Body Mobile Manipulation Problem

Picture a robot that must navigate across a room, identify a cup on a table regardless of lighting conditions, approach from the right angle, grasp it precisely, and place it exactly where requested — all in one continuous, coordinated motion. This is whole-body mobile manipulation: simultaneously controlling a mobile base, torso, dual arms, and head as a unified system.

Three intertwined challenges make this genuinely hard:

High-dimensional action space: A whole-body mobile manipulator can have 25–37 degrees of freedom. Generating coherent, safe joint trajectories across all DOF simultaneously — where base, torso, and arms all move in sync — is a fundamentally harder coordination problem than controlling a fixed-arm robot.

Appearance variation in deployment: Robots trained in a lab will encounter different lighting, object colors, spatial layouts, and entirely new rooms in the real world. A policy that memorizes visual appearance rather than understanding geometry and semantics will break the moment conditions change.

The geometry–semantics tradeoff: Precise manipulation requires knowing where objects are in 3D space (geometry), but recognizing what they are despite appearance changes requires semantic understanding. These two modes of perception pull in different directions — 3D methods are geometrically precise but brittle to color/texture changes; 2D foundation models are semantically robust but lack spatial grounding.

DSPv2 (Dense Policy version 2), accepted at ICRA 2026 from the University of Hong Kong, Astribot, and Tsinghua University, directly addresses all three challenges with a modular architecture that fuses 3D spatial reasoning with multi-view 2D semantic perception.

Paper: DSPv2: Improved Dense Policy for Effective and Generalizable Whole-body Mobile Manipulation — Yue Su, Chubin Zhang, Sijin Chen et al., ICRA 2026
GitHub: Selen-Suyue/DSPv2
Project page: selen-suyue.github.io/DSPv2Net

DSPv2 system overview — source: DSPv2Net project page

What Is DSPv2?

DSPv2 is an improved version of Dense Policy (DSP) — a robot learning method using bidirectional autoregressive action generation, originally published at ICCV 2025 for fixed-arm manipulation. DSPv2 extends this paradigm to whole-body mobile manipulation and introduces a principled solution to the generalization problem.

Three core contributions:

1. Effective 2D-3D feature fusion: Rather than committing to either 3D point clouds (DP3) or 2D RGB-only (DP), DSPv2 runs both pathways in parallel and aligns them through a Q-Former mechanism — preserving the strengths of each without inheriting their weaknesses.

2. First systematic generalization study for whole-body manipulation: DSPv2 explicitly evaluates across four generalization conditions: lighting changes, object color changes, spatial rearrangement, and entirely new scenes. Most prior work reports only in-distribution performance.

3. Dense Action Head for whole-body control: A bidirectional autoregressive transformer generates 37-dimensional action sequences faster than diffusion-based methods while maintaining trajectory coherence.

Technical Architecture in Depth

1. Multi-View Camera Setup

The Astribot-S1 robot — DSPv2's experimental platform — carries three classes of cameras:

• Head camera (Orbbec Femto Bolt): RGB + depth, wide-field workspace overview
• Torso camera (Orbbec Gemini 335): mid-range perspective for workspace coverage when the head rotates
• Wrist cameras (Intel Realsense D401 × 2): close-range, high-resolution observation during contact

The robot state is a 37-dimensional merged pose:

• Chassis: 3 DOF (x, y, heading)
• Torso: 1 DOF (lift)
• Left arm: 7 DOF
• Right arm: 7 DOF
• Left + right gripper: 1 DOF each
• Head: 2 DOF

During training, robot state undergoes 30% probabilistic masking — randomly zeroing out portions of proprioception. This prevents the policy from over-relying on joint feedback and forces it to reason more from visual input, improving robustness when sensors are noisy in deployment.

2. 3D Branch: Sparse Encoder on Color-Stripped Point Clouds

The head depth camera generates dense point clouds. DSPv2 makes a deliberate choice: strip color information before processing the point cloud.

The reasoning is sound: color in a point cloud depends on lighting conditions, object surface properties, and camera calibration — all of which vary continuously across deployments. A 3D encoder trained on colored point clouds will fail the moment object color or lighting changes. By using uncolored point clouds (XYZ coordinates only), the 3D branch learns pure spatial geometry: positions, distances, surface shapes — properties invariant to appearance changes.

The point cloud is transformed into the robot base frame before encoding. This ensures positional consistency regardless of which direction the robot is facing.

A Sparse 3D Encoder (sparse convolution architecture similar to MinkowskiEngine) processes the voxelized point cloud (~1cm voxel size) and outputs geometric features $F_{3D}$ paired with the 3D coordinates of each feature voxel.

3. 2D Branch: DINOv2 With LoRA Fine-Tuning

In parallel, RGB images from all four cameras pass through DINOv2-base — Meta AI's vision foundation model pre-trained on 142 million images with self-supervised learning. DINOv2's features are semantically rich: images of the same object type, even under different lighting or viewpoint, produce similar feature vectors.

The core challenge: DINOv2 was pre-trained on natural images, not robotics data. Fully frozen, its features are semantically strong but miss domain-specific manipulation cues. Fully fine-tuned, it risks losing the generality that makes it valuable.

The solution is LoRA (Low-Rank Adaptation) — inserting trainable low-rank matrices into DINOv2's attention layers. Only ~1% of parameters are updated during training, preserving the general representation while adapting to the robotics domain.

The paper validates this empirically (Table III): DINOv2 + LoRA achieves 80% success in novel scenes versus 60% for frozen DINOv2 — a 20-point improvement from a minimal modification.

4. Q-Former: The Geometry–Semantics Bridge

This is DSPv2's most elegant contribution. The Q-Former holds 300 learnable query tokens that must bridge two modalities: the 3D branch knows where but not what; the 2D branch knows what but lacks precise spatial grounding.

DSPv2's solution: use the 3D coordinates of each feature voxel as positional embeddings for the query tokens. A query token carrying positional embedding corresponding to 3D point $P$ will attend, via cross-attention, to the 2D feature map region where point $P$ is projected across each camera view.

The full flow:

1. Sparse 3D Encoder outputs $(F_{3D}, \text{coords}_{3D})$ — features with associated 3D positions.
2. 3D coordinates are embedded as positional encodings.
3. 300 learnable query tokens + positional encodings cross-attend into multi-view 2D feature maps.
4. Output: 300 spatially-grounded semantic tokens — each token knows both where it is in space (from positional embedding) and what is there semantically (from DINOv2 features).

This design keeps the two branches fully independent in the forward pass — no shared weights, no forced common feature space — while achieving alignment through the Q-Former's cross-attention. The geometric branch and semantic branch can each specialize without compromise.

5. Dense Action Head: Generating Coherent Whole-Body Actions

The 300 Q-Former tokens combined with robot state feed into the Dense Action Head, inherited from the original Dense Policy work.

Instead of a diffusion model (requiring many denoising steps, adding latency and variance), Dense Head uses a bidirectional autoregressive transformer with a coarse-to-fine generation strategy:

• Forward pass (coarse): Generate sparse keyframe actions outlining the overall trajectory from current state to goal state.
• Backward pass (fine): Fill in intermediate steps, conditioning on both preceding and following keyframes simultaneously.

Bidirectional attention directly addresses error accumulation: in standard unidirectional autoregressive generation, errors at step $t$ propagate and amplify through $t+1, t+2, \ldots$. With bidirectional context, each intermediate step is anchored by future keyframes, preventing trajectory drift.

Final output: 37-dimensional action sequences for a chunk of 20 steps.

Installation and Setup

Requirements

• GPU: NVIDIA with ≥16GB VRAM (DINOv2-base + Sparse 3D Encoder)
• CUDA 11.8+
• Python 3.9+
• RAM: ≥32GB (point cloud processing is memory-intensive)

Environment Setup

# Clone DSPv2
git clone https://github.com/Selen-Suyue/DSPv2.git
cd DSPv2

# Create dedicated conda environment
conda create -n dspv2 python=3.9 -y
conda activate dspv2

# Install PyTorch with CUDA
pip install torch==2.1.0 torchvision==0.16.0 \
    --index-url https://download.pytorch.org/whl/cu118

# Install remaining dependencies
# Full instructions: assets/docs/INSTALL.md
pip install -r requirements.txt

Robot-Specific Configuration

Before anything else, adapt the constants to your robot:

# dataset/constants.py
ROBOT_DOF = 37          # Your robot's action dimensionality
CONTROL_HZ = 20         # Control frequency in Hz
ACTION_CHUNK = 20       # Actions generated per inference
CAMERA_NAMES = ['head', 'torso', 'wrist_left', 'wrist_right']
DATA_DIR = '/path/to/your/data'

# utils/constants.py
JOINT_LIMITS_MIN = [...]   # Per-joint lower limits (rad or m)
JOINT_LIMITS_MAX = [...]   # Per-joint upper limits
POINT_CLOUD_VOXEL = 0.01   # Voxel size in meters

If you are not using Astribot-S1, also adjust the input/output layer dimensions in policy/policy.py to match your robot's DOF. The Q-Former and Dense Head are robot-agnostic; only the first and last projection layers need modification.

Data Preparation

DSPv2 uses an RH20T-compatible data format. Each episode is a directory:

episode_000/
├── metadata.json          # Task, robot, calibration info
├── cam_head_rgb/          # RGB frames from head camera (174 frames)
├── cam_head_depth/        # Depth maps from head camera
├── cam_torso_rgb/         # RGB frames from torso camera
├── cam_wrist_left_rgb/    # RGB frames from left wrist
├── cam_wrist_right_rgb/   # RGB frames from right wrist
└── robot_state.h5         # HDF5: joint positions, TCP coords, gripper state

Once you have collected 100 demonstrations per task via VR teleoperation, preprocess them:

# Step 1: Preprocess — voxelize point clouds, extract features
python dataset/preprocess_data.py \
    --data_dir /path/to/raw_episodes \
    --output_dir /path/to/processed \
    --task pick_and_place \
    --voxel_size 0.01

# Step 2: Compute normalization statistics — REQUIRED before training
# Calculates 5th–95th percentile across all actions, saves to dataset/pose.json
python dataset/compute_normalization.py \
    --processed_dir /path/to/processed \
    --task pick_and_place

# Step 3: Verify dataset integrity
python utils/hdf5_view.py --file /path/to/processed/episode_000/robot_state.h5

The normalization step is often skipped but is critical: DSPv2 normalizes all actions to [-1, 1] before training. Without it, training either diverges or converges extremely slowly.

Training

conda activate dspv2

# Train with default config
bash train.sh

# Or with explicit arguments:
python train.py \
    --task pick_and_place \
    --data_dir /path/to/processed \
    --batch_size 32 \
    --lr 1e-4 \
    --epochs 200 \
    --use_lora True \
    --lora_rank 16 \
    --mask_prob 0.3 \
    --checkpoint_dir checkpoints/

Key training hyperparameters:

Monitor training:

tensorboard --logdir logs/ --port 6006

Watch three losses:

• action_loss: Primary loss, must decrease steadily.
• 3d_feat_loss: 3D branch quality, important for spatial understanding.
• kl_loss: Q-Former regularization. Sudden increase = overfitting signal.

Models typically converge around 80–120 epochs with 100 demonstrations.

Inference

conda activate dspv2

# Evaluate on real robot
python eval.py \
    --checkpoint checkpoints/best_model.pt \
    --task pick_and_place \
    --num_episodes 10 \
    --record_video True

# Or use the preset evaluation script
bash eval.sh

Watch DSPv2 performing pick-and-place on Astribot-S1:

DSPv2 uses action chunking: rather than running the network once per control step, the policy generates a 20-step action chunk and executes it open-loop. This reduces network overhead and produces smoother trajectories since all steps in the chunk share the same observation context.

Experimental Results

Task Performance (In-Distribution)

All results on Astribot-S1, 100 demonstrations per task:

Sort achieves 100% because it requires categorical recognition and placement but not high-precision grasp. Cart pushing reaches 90% because the task relies on whole-body pushing force rather than precise grasping.

Generalization Results — The Key Contribution

The most revealing number: DP3 drops to 10% under object color changes. DP3 uses colored point clouds — when object color changes, the point cloud looks different, and the policy fails to recognize the object. DSPv2 achieves 80% in the same condition by decoupling geometry (uncolored point cloud) from semantics (DINOv2 RGB features). The two branches are explicitly designed so that color changes affect only the 2D branch while the 3D branch remains stable — and DINOv2's pre-trained representations handle color variation naturally.

Ablation Studies

Each component contributes: LoRA adds 20 points, Q-Former adds ~15 points over naive concatenation, Dense Head matches diffusion accuracy while being 3× faster.

Comparison With Related Methods

DSPv2 is the only method in this comparison that simultaneously achieves: whole-body control, strong generalization, and fast inference. For projects that need a whole-body mobile manipulation baseline with real-world generalization, it is currently the strongest option with a public codebase.

To understand the dense model family more broadly, see Dense Models in Robotics: Comprehensive Comparison. For practical mobile manipulation with LeRobot, see VLA for Mobile Manipulation with LeRobot.

Limitations and Future Directions

DSPv2 is transparent about its boundaries:

Cross-robot transfer: The policy trained on Astribot-S1 degrades significantly when deployed on robots with substantially different kinematics or DOF configurations. Cross-embodiment generalization for whole-body manipulation remains an open problem.

Data collection overhead: 100 demonstrations per task through VR teleoperation is manageable but not trivial. Scaling to 20+ tasks requires significant infrastructure investment.

Real-time inference constraints: Dense Head is faster than diffusion but still requires a capable GPU to maintain ≥10Hz control frequency for 37-DOF whole-body control.

Directions the authors suggest: Integrating world models to generate synthetic demonstrations, reducing dependency on physical teleoperation, and extending the cross-embodiment transfer capability.

For a broader view of whole-body VLA training pipelines, see Whole-Body VLA Training Pipeline: From Data to Deployment. For RL-based dense reward approaches, see ProcVLM: Dense Reward VLA with Reinforcement Learning.

Conclusion

DSPv2's strength is not a single breakthrough technique but a well-designed combination of independently sound choices: uncolored point clouds for geometric robustness, DINOv2 + LoRA for semantic generalization, Q-Former for flexible alignment, Dense Head for fast inference. Each component solves exactly one problem. The result is a modular architecture that can be adapted to different robot platforms by adjusting input/output dimensions while keeping the core perception and action generation logic unchanged.

For roboticists building policies for whole-body mobile manipulators who need demonstrated generalization beyond lab conditions, DSPv2 is currently the most principled starting point with a public implementation.

Related Posts

• Dense Models in Robotics: Comprehensive Comparison
• Whole-Body VLA Training Pipeline: From Data to Deployment
• ProcVLM: Dense Reward VLA with Reinforcement Learning