SimpleMemVLA: Native Video Memory for Long-Horizon VLA

How does a robot remember what it did two minutes ago in order to make the right decision now? This is not a compute problem — it is a memory design problem. And most existing approaches get the design fundamentally wrong.

SimpleMemVLA: A Simple but Effective Native-Video Memory for Vision-Language-Action Models (arXiv:2609.05533, September 2026) by Cheng Yin, Wang Xu, Junpeng Yang and colleagues delivers a surprising answer: no dedicated memory module is needed at all. Feed the right data in the format the backbone was already pretrained to process, and its native self-attention does the rest.

The Problem: Why Long-Horizon Manipulation is Hard

Consider a task where a robot places three blocks into colored boxes in some order, then must remove them in the exact reverse order. The critical information — which block went in first — disappeared from the camera frame minutes ago. The current observation contains no trace of it.

This is the core of partial observability: the policy needs information from the past, but the current observation does not contain it. Standard VLA models only see the current frame or a short recent window. They are completely blind to long-horizon history.

Four benchmarks measure this capability systematically:

• RMBench — 9 tasks on bimanual Aloha-AgileX robot; requires tracking object identity and placement order across many steps
• RoboMME — 16 tasks on a Panda arm; tests sequential, spatial, and semantic memory
• MIKASA-Robo — wrist-camera setup with dense short-horizon memory requirements
• RoboMemArena — the longest horizon; up to 126-second history windows

Why Prior Memory Mechanisms Fail

Three families of memory mechanisms have been tried before SimpleMemVLA:

Retrieval-based memory

Store "important" observations in a memory bank; retrieve them at decision time. Core flaw: you must decide what is important at write time — before you know what a future decision will require. This is the write-time commitment problem.

Token compression

Compress the history into a fixed token budget (e.g., one token per frame). Core flaw: information loss is irreversible. The compressor cannot anticipate which details a future step will need.

Recurrent state (RNN-style)

Maintain a hidden state that summarizes history, updated at each step. Core flaw: the state is lossy by design — each update can overwrite prior context, which is precisely where ordering and causal information disappear.

SimpleMemVLA's ablation on RoboMME — same Qwen3.5-4B backbone, same training data, only the memory interface changes:

The gap — 88.3% vs. 20–31% — cannot be explained by backbone capacity or data volume. It is entirely attributable to the memory interface design.

The Core Insight: VLMs Already Know How to Read Video

Qwen3.5-4B was pretrained on large quantities of timestamped video. It already knows how to process video — that is exactly the format it was trained on from the beginning.

SimpleMemVLA's insight: do not teach the backbone a new way to remember using a new module — give it data in the format it already knows how to process.

Instead of compressing, retrieving, or maintaining recurrent state:

1. Keep raw observations with their temporal ordering and plaintext timestamps, exactly as video
2. Feed the full history clip directly into the backbone's video channel
3. Let the backbone's native self-attention decide which frames matter at each decision point

No premature commitment to what is important. Evidence selection is deferred to decision time. A human expert reviewing a video does not summarize it before knowing the question — they read through looking for relevant clues. SimpleMemVLA lets the backbone do exactly that.

Architecture

SimpleMemVLA has three components connected in sequence:

Component 1 — Video History Input

The model maintains a history window V_t = (frame_{t−Ks}, …, frame_{t−s}, frame_t):

• 60 seconds at 2 fps = 120 frames
• Each frame prefixed with a plaintext timestamp in Qwen3.5-4B's exact video pretraining format (e.g., [t=42.3s])
• A 60-second window uses only approximately 5,600 tokens of the 262,000-token context — enough room for roughly 45 minutes of history

The main camera delivers video history through the backbone's video channel. The wrist camera contributes only the current frame through the image channel — past vs. present separated by modality, no special encoding required.

Component 2 — Sub-task Generation (History-to-Action Bridge)

Given the full video history, the backbone generates a brief description of the current sub-task:

• At most 64 tokens — no chain-of-thought, no extended reasoning
• Supervised with token-level cross-entropy against ground-truth sub-task labels
• The hidden states and token embeddings from this span are the only channel through which history information reaches the action head

This creates a meaningful bottleneck: the model compresses what it understood from video history into ≤64 tokens — but this compression happens after reading the full history, not before. Because the same span masking applies at training and inference time, there is no train/rollout conditioning drift.

Component 3 — Flow-Matching Action Head

A DiT (Diffusion Transformer) with approximately 0.9 billion parameters:

• Conditioning set C_t = [token embeddings + hidden states from sub-task span] + [normalized proprioceptive token]
• Inference: 10 deterministic Euler integration steps
• Loss: standard flow-matching — E[‖v_φ(A^τ, τ | C_t) − (ε − Ā_t)‖²]

No exotic architecture here — this is a standard flow-matching action head. The only difference from typical VLA heads is that conditioning comes from sub-task hidden states rather than from a single current frame. See A1-VLA Truncated Flow Matching for a deeper look at this action head design pattern.

Streaming Inference: Maintaining Single-Frame Latency

The robot executes an action over roughly 0.96 seconds. During that time, SimpleMemVLA prefills the shared video prefix rather than waiting idle.

Since consecutive decisions share nearly all of their video history prefix (only the newest frames differ), the system:

1. Caches the KV state of the already-computed prefix
2. On the next decision, only computes the newly added frames
3. Produces output that is bit-identical to full recomputation — no accuracy loss

Measured latency (60-second history):

Scaled to a 45-minute history, streaming reduces latency from 32.1 s to 1.18 s — a 27× speedup while preserving exact outputs.

Benchmark Results

The most striking result: on RoboMME, SimpleMemVLA reaches 88.3% — above the GT-oracle baseline at 84.1% (the oracle receives ground-truth environment state). This suggests the model learns to exploit video history more effectively than directly observing the ground-truth answer — a sign it has genuinely internalized a useful memory-reading strategy.

The +43.7 pts improvement on RoboMME is the largest ever reported on that benchmark, and it comes from changing the memory interface, not the backbone or training data.

Installation and Usage

Requirements

Python 3.10
CUDA GPU with flash-attention support
PyTorch 2.4.1 (required for SAPIEN simulator compatibility)

Environment setup

Each benchmark requires a separate conda environment due to SAPIEN version conflicts across simulator families:

conda create -n simplememvla-rmbench python=3.10 -y
conda activate simplememvla-rmbench
pip install -r requirements.txt

# Install fast-path kernels (required for training and evaluation)
bash scripts/install/install_fast_path.sh

# Install benchmark-specific simulator
bash scripts/install/install_rmbench_sim.sh
# Replace "rmbench" with: robomme | mikasa | robomemarena | libero

Download pre-trained checkpoints

# From Hugging Face
huggingface-cli download yinchenghust/simplememvla_rmbench \
  --local-dir checkpoints/simplememvla_rmbench

# From ModelScope (faster in China)
modelscope download yinchenghust/simplememvla_rmbench \
  --local_dir checkpoints/simplememvla_rmbench

Checkpoints are self-contained: they include model weights, config, processor, and dataset statistics. The exact training pipeline can be reconstructed from config.json.

Training from scratch

bash scripts/train.sh rmbench
# Options: rmbench | robomme | mikasa | robomemarena | libero

Override defaults via environment variables:

HISTORY_VIDEO_SEC=60 \
GPUS_PER_NODE=8 \
PER_DEVICE_BATCH=2 \
  bash scripts/train.sh robomme

Key training settings:

• Optimizer: AdamW with split learning rates for backbone vs. action head
• Schedule: 1,000-step linear warmup + cosine decay
• Distributed: DeepSpeed ZeRO-2 by default
• History window: 60 s (override via HISTORY_VIDEO_SEC)

Closed-loop evaluation (with simulator)

CHECKPOINT=checkpoints/simplememvla_rmbench \
NUM_GPUS=8 \
  bash scripts/eval_rmbench.sh

Open-loop evaluation (no simulator needed)

Faster check for action L1 loss and sub-task prediction accuracy:

bash scripts/eval_openloop.sh rmbench

Real-Robot Deployment

The authors deployed SimpleMemVLA on a physical dual-arm robot for the Cover Blocks task:

• Robot places covers over three blocks in sequence, then must remove them in the exact reverse order (red → green → blue)
• The correct removal order cannot be inferred from the current spatial arrangement — it can only be inferred from video history captured 52–98 seconds earlier
• Three autonomous rollouts across three different block layouts

Real-robot results:

• Cover Blocks: 58.3% success rate
• Put Back Block: 70.0% success rate

Causal intervention analysis confirmed that the frames driving behavior are 52–98 seconds in the past — the model genuinely reads and uses long-horizon history rather than relying on any spatial shortcut from the current observation.

Comparison with Related Memory Approaches

SimpleMemVLA also supports Sliding Window Attention (SWA) to handle histories longer than the base context window without unbounded cache growth — useful for deployment under GPU memory constraints.

For a deep look at recurrent memory alternatives, see μVLA: Recurrent Memory Token for Partially Observable Manipulation. For a world-model approach, see MemoryVLA++: Memory and World Model for VLA.

The Design Lesson

SimpleMemVLA teaches an important principle about AI system design: do not build a new abstraction when an existing component already has the capability — just use it correctly.

Retrieval, compression, and recurrent state were all answers to the question "how do we help an older backbone handle long history?" When the backbone is a modern VLM pretrained on video, the question is obsolete — the backbone already handles video natively. What remains is to deliver data in the right format, not to build an adapter for the wrong format.

This is why SimpleMemVLA — architecturally simpler than every system it beats — outperforms them by margins that would be considered impossible before this paper. See BridgeVLA+: Spatio-Temporal Memory for 3D Manipulation for a different angle on solving long-horizon memory through spatial representations.

Conclusion

SimpleMemVLA demonstrates that simplicity in the right place is good research: by eliminating dedicated memory modules and leveraging Qwen3.5-4B's existing video-processing capability, it achieves state-of-the-art on all four memory-centric benchmarks — with the largest gains ever seen on RoboMME (+43.7 pts) and MIKASA-Robo (+29.6 pts). Streaming inference with prefix caching keeps decision latency near that of a single-frame VLA, even over 45-minute histories.

As VLM backbones continue to improve, many "problems" in robot learning will be solved by giving them the right input rather than adding specialized modules. SimpleMemVLA is an early, compelling demonstration of that shift.

Paper, code, and checkpoints are publicly available at github.com/hongzefu/SimpleMemVLA — MIT license.

Related Posts

• μVLA: Recurrent Memory Token for Partially Observable Manipulation
• MemoryVLA++: Memory and World Model for VLA
• BridgeVLA+: Spatio-Temporal Memory for 3D Manipulation