RynnVLA-Latent: 3D Actions from Sample Data

A video can show someone picking up a cup without telling a robot how far to move its motors. Even robot datasets are difficult to combine: one controller expects joint positions, another expects an end-effector pose, and a dual-arm platform has a different action dimensionality. RynnVLA-Latent introduces an intermediate representation that lets a policy learn from heterogeneous interaction videos before learning the commands of a particular robot.

This tutorial starts with the synthetic data included in the repository. You will learn the architecture, install the environment, validate a dataset, run smoke training, and understand what must be added for robot post-training and inference. You do not need a physical robot for the initial steps. However, Stage-1 weights are not executable robot policies: closed-loop evaluation requires Stage 2 and action-labeled demonstrations.

The primary references are the RynnVLA-Latent paper, a preprint by Haoyu Zhao and colleagues dated October 25, 2026, and the official repository. Commands were checked against the public source on November 1, 2026. This article does not claim that its GPU training or benchmark reproduction has been executed.

1. Learn motion first, align control afterward

VLA stands for Vision-Language-Action. Such a model receives images, a language instruction, and sometimes robot state, then predicts actions. The difficult part is defining the action target. Placing a cup on a table means something similar visually for a human hand and a robot gripper, but their command spaces are different.

RynnVLA-Latent separates policy learning into two stages. Stage 1 predicts latent actions, vectors describing transitions between observations, from images and language. Stage 2 uses demonstrations from the target embodiment to change the prediction target into native robot commands. Robot-action data is still required for this final alignment; unlabeled video does not remove that requirement.

The label generator is RynnLAM, a self-supervised latent-action model. It observes a current frame and a future frame and encodes their transition. The policy subsequently learns to predict that descriptor using information available when making a decision. The future frame belongs to offline supervision, so deploying the trained policy does not require access to future observations.

For background on observations, episodes, and policy training, see our hands-on LeRobot introduction.

2. What makes the latent action representation 3D-grounded?

An egocentric camera can turn while the person wearing it stays in place. Pixel motion alone can therefore confuse viewpoint changes with manipulation. RynnLAM adds geometric supervision to encourage its representation to capture physical motion rather than simply visual differences.

Source: Figure 3 in the authors' RynnVLA-Latent paper.

The exported ktoken_zcam descriptor has 608 dimensions:

RynnLAM builds on Depth Anything 3. Its encoder computes a gated temporal difference between features from the two frames, making changes explicit while suppressing irrelevant static appearance. Learned readout tokens and motion queries produce the descriptor components.

Five training objectives provide complementary constraints. Residual feature reconstruction predicts feature changes instead of reproducing a complete image. Dynamic 3D flow uses pseudo-labels derived from estimated depth, camera poses, and optical flow, with camera-motion compensation. A contrastive feature objective encourages flow-warped source features to resemble the target more than the unchanged source. A camera-pose objective trains the camera latent to retain pose information. Gradient reversal discourages the global action latent from revealing that pose.

There is an important qualification: camera decoupling specifically targets the global action latent. The full 608-dimensional descriptor still includes the camera latent, so it should not be described as completely camera-invariant. Geometric pseudo-labels are estimates, not perfect measurements, and can be unreliable under occlusion or poor depth estimation.

3. How the VLM and action expert work together

The policy uses a Mixture-of-Transformers architecture. RynnBrain turns language and multi-view images into a VLM prefix. A narrower action expert interacts with that prefix through attention at each layer while retaining separate output projections and feed-forward pathways. The prefix remains causal; the action suffix attends to the prefix and uses bidirectional attention within its own tokens.

Source: Figure 4 in the authors' RynnVLA-Latent paper.

The 2B and 4B names describe backbone variants. Including the action expert, the paper reports approximately 2.51B and 5.04B total policy parameters. The expert has a hidden width of 768. The observation interface provides six camera slots, masking missing views out of attention and loss. This accommodates datasets with different camera availability, although it does not guarantee robustness to every unseen viewpoint.

Training uses flow matching. Informally, a clean target chunk is mixed with Gaussian noise, and the model learns a velocity field that describes how to recover the target given images and language. At inference, sampling starts from noise and follows that field using Euler integration. The VLM prefix is computed once and reused across the integration steps.

Stage 1 predicts six latent targets per chunk. Its configuration also contains action_chunk_size: 30, which should not be interpreted as thirty separate 608-dimensional descriptors. Read it alongside latent_action_chunk_size: 6 and latent_action_stride: 5. The Stage-2 V5 preset predicts native action chunks of length 10, introduces proprioception, and changes the output projection. It does not attach a separate latent-to-motor decoder after the policy.

For a comparison with another use of video representations, read VLA-JEPA and latent world models.

4. Install in a dedicated environment

The training environment targets Linux with NVIDIA CUDA. The current requirements.txt references Python 3.10, PyTorch 2.7.1 with CUDA 12.6, Transformers 5.2.0, and a Linux x86_64 flash-attn wheel. That compiled wheel has ABI requirements, so the same installation should not be assumed to work on ARM64, macOS, or arbitrary Python/CUDA combinations.

Download a source archive into a separate learning directory where RynnVLA-Latent-main does not already exist:

curl -fL https://github.com/alibaba-damo-academy/RynnVLA-Latent/archive/refs/heads/main.tar.gz -o rynnvla-latent-main.tar.gz
tar -xzf rynnvla-latent-main.tar.gz
cd RynnVLA-Latent-main
conda create -y -n rynnvla python=3.10
conda activate rynnvla
python -m pip install -r requirements.txt

The main archive can change. For a reproducible experiment, record the exact source revision you downloaded, the model identifier and revision, and all configuration files. Two runs using the same model name do not necessarily use identical weights or training recipes.

Check the environment before starting training:

python -c "import torch; print(torch.__version__); print(torch.cuda.is_available())"
python -c "import rynnvla, rynnlam; print('packages OK')"
python -c "import flash_attn_2_cuda; print('flash-attn extension OK')"

Importing the compiled extension catches a missing or incompatible wheel more directly than importing only the Python package. If a downloaded wheel has zero bytes, fix the transfer first. If an import reports undefined symbols, check the Torch, CUDA, Python, and ABI combination. Keep these changes inside the dedicated environment so another training project remains reproducible.

5. Start with the bundled synthetic datasets

Two small synthetic corpora are included to exercise the data and training paths:

Begin with the CPU data check, which does not need backbone weights:

bash scripts/smoke.sh --only data

This loads samples through LatentPretrainDataset; it does more than check that files exist. The script verifies decoded images, text, slot masks, and finite latent targets whose last dimension is 608. Look for [verify] OK. Read the sample count from the actual log rather than calculating it by multiplying episodes and latent descriptors, because dataset indexing determines usable windows.

The bundled mixture in data/sample/sample_mixture.json looks like this:

[
  {
    "data_type": "LatentPretrainDataset",
    "data_path": "data/sample/build/index_c6.npz",
    "latent_stats_path": "data/sample/build/latent_stats.json",
    "latent_chunk": 6,
    "latent_step_seconds": 0.25,
    "dataset_weights": {"SampleToy": 1.0}
  }
]

The data_path points to an index, not an arbitrary video folder. The latent statistics must describe the corresponding corpus. When replacing the samples, preserve alignment among video frames, latent timestamps, indices, and normalization statistics. The launcher does not download datasets, extract latent actions, or build an index. JSON paths also do not expand environment variables automatically.

6. Run RynnLAM smoke training

After the dataset check, run:

bash scripts/smoke.sh

By default, the wrapper checks the data and runs two RynnLAM optimizer steps. The rynnlam/configs/smoke.yaml recipe uses a randomly initialized encoder and batch size 1, without external DA3 weights. All five loss branches are enabled to exercise training, checkpointing, and internal evaluation.

Checkpoints are written below runs/smoke_lam. Confirm a successful process exit, finite losses, and a saved checkpoint. A small loss on synthetic scenes is not evidence that the encoder understands real manipulation. This recipe is an installation and execution check, not a miniature benchmark reproduction.

To invoke the trainer directly instead of using the wrapper:

python scripts/train_rynnlam.py --config rynnlam/configs/smoke.yaml

Real RynnLAM training requires compatible DA3-Large encoder weights, including the expected QK-normalization tensors. The paper describes 36,000 steps with the encoder frozen followed by 20,000 steps with it unfrozen. These two RynnLAM training phases are distinct from VLA Stage 1 and Stage 2. Use explicit run names so that a latent-model checkpoint is never mistaken for a policy checkpoint.

7. Smoke the VLA, then prepare real pretraining

The VLA smoke path needs a local RynnBrain-2B checkpoint, or a supported Qwen3-VL-family checkpoint, for tokenizer and processor assets. Replace the example path with your complete local directory:

bash scripts/smoke.sh --src /path/to/RynnBrain-2B

The script creates a tiny, approximately 44M-parameter random-weight backbone and trains for two steps. It does not fine-tune the complete 2B model. Without --src and without an existing tiny backbone, it records the VLA stage as skipped. A successful wrapper exit must therefore be read together with its stage summary.

For actual Stage-1 pretraining, prepare local RynnBrain weights and a valid mixture referencing a corpus already labeled by RynnLAM. The following command is a template for assets you must supply, not an automatic dataset or weight downloader:

python -m rynnvla.api.launch \
  --stage stage1_2b \
  --model-path models/RynnBrain-2B \
  --data-mixture configs/data_latent_pretrain.local.json \
  --output-dir runs/stage1-2b \
  --nproc-per-node 8

Append --dry-run to inspect the generated command without starting training. The reference configuration uses micro batch 4, gradient accumulation 4, and eight ranks, giving global batch 128. Reducing the GPU count does not automatically preserve that batch or adjust the learning rate. Gradient checkpointing and bf16 help memory usage, but this guide does not promise a minimum VRAM capacity: images, sequence lengths, optimizer state, and the selected model all matter.

8. Post-train for an actual robot action space

Stage 2 can start from your own Stage-1 export or an author-released Stage-1 checkpoint. Download complete model and processor assets first; runtime loading uses local_files_only=True.

For your own Stage-1 run, an EMA export at step 60,000 would use:

mkdir -p models
python -m rynnvla.api.export_checkpoint \
  --checkpoint-dir runs/stage1-2b/checkpoint-60000 \
  --output-dir models/stage1-2b-ema \
  --weights ema --format safetensors

The checkpoint must exist and include ema_model.bin. Selecting EMA does not silently fall back to live weights. The destination export directory must not already exist. Next, prepare LIBERO demonstrations matching the layout in configs/data_libero_joint40.example.json, create a local mixture with valid paths, and launch:

python -m rynnvla.api.launch \
  --stage stage2_2b \
  --model-path models/stage1-2b-ema \
  --data-mixture configs/data_libero_joint40.local.json \
  --output-dir runs/stage2-2b \
  --nproc-per-node 8

This starts a new optimizer run. It is not a resume of Stage 1. Use --resume only to continue the same stage in the same output directory. The synthetic latent corpus cannot replace action-labeled Stage-2 demonstrations. Camera mapping, state schema, action normalization, and rotation representation must match the embodiment, particularly when moving between single-arm and dual-arm control.

9. Check offline inference before closed-loop evaluation

After Stage 2, export a real checkpoint into a Hugging Face model directory. This example assumes that training saved a checkpoint at step 2,000:

mkdir -p exports
python -m rynnvla.api.export_checkpoint \
  --checkpoint-dir runs/stage2-2b/checkpoint-2000 \
  --output-dir exports/stage2-2b \
  --weights model --format safetensors
python -m rynnvla.api.predict --help

The offline predictor reads JSON containing instruction, state, front, and wrist. The final two fields point to image files relative to the JSON file. Supply a state vector matching the LIBERO schema rather than inventing arbitrary values. The output is a .npy array of raw OSC commands with shape (T, 7). Use the CLI's --sample and --output arguments to inspect one prepared observation before connecting a simulator.

Closed-loop evaluation additionally requires upstream LIBERO, robosuite/MuJoCo, BDDL tasks, and offscreen rendering. These are not installed by this repository. Once the simulator dependencies and environment compatibility are resolved, run:

python -m rynnvla.api.eval_libero \
  --model-path exports/stage2-2b \
  --suite libero_spatial --task-id 0 --episodes 1 \
  --seed 7 --output-dir results/libero-spatial-task0 \
  --replan-steps 10 --denoising-steps 10 --save-frames

The adapter owns image orientation, state conversion, and rot6d-to-OSC decoding. Do not transform images or rescale commands a second time before handing them to it. A single episode checks integration. A benchmark claim requires the full protocol, sufficient episodes, identified checkpoints, seeds, and explicit rollout and replanning budgets.

10. Reported results and what they establish

Source: the Training Data figure on the official project page.

The paper uses 30.1 thousand hours of video across 25 datasets, producing approximately 745.52 million descriptors. That research-scale corpus is fundamentally different from three synthetic episodes. The following are author-reported results for the post-trained 4B policy, not measurements from this tutorial:

Sources: the paper's experiments and official project results. Zero-shot evaluation on LIBERO-Plus does not mean that the policy never saw robot-action training. The paper also evaluates six real-world tasks on two platforms, but those observations do not establish the same performance on a different robot, object set, or camera setup.

Practical limitations include geometric pseudo-label quality, the cost of labeling and training, the continued need for target-embodiment demonstrations, and the gap between synthetic smoke tests and manipulation success. The repository provides policy-side support for RoboTwin and VLABench, while their complete simulators, runners, and corpora are not redistributed. One command here cannot reproduce every reported table.

A useful next experiment keeps the sample corpus as an installation check, then chooses one embodiment and one task with reliable demonstrations. Record dataset validation, training losses, offline predictions, and closed-loop outcomes separately. Expand to additional tasks or hardware after those layers have been verified. For another approach to future prediction in policy learning, see PLAW with V-JEPA 2 on LIBERO.

Related Posts

• A hands-on LeRobot introduction
• VLA-JEPA and latent world models
• PLAW: future prediction for LIBERO VLA policies