FM-VLA: Force Memory Tokens for Non-Markovian Contact-Rich Manipulation

Imagine a robot that presses a button exactly three times and stops — not because it sees a counter on a screen, but because it remembers each press through the force spikes it felt. That is the core capability FM-VLA delivers, and it does so while adding only 3ms of inference overhead on top of its base model.

The paper FM-VLA: Force-based Memory for Vision-Language-Action Models in Contact-Rich Manipulation by researchers from Tsinghua University and Microsoft Research (July 2026) asks a fundamental question: why do we limit VLA memory to visual history, when robots have force sensors that directly capture interaction events?

The Root Problem: VLA Models Are Goldfish

Most state-of-the-art VLA models — RT-2, π0, π0.5, OpenVLA — operate under the Markovian assumption: at each step, the model sees only the current observation (image + language instruction) and outputs an action. No memory of the past.

For simple pick-and-place tasks, this is fine. But real-world manufacturing and service robotics require more:

• Counting repeated actions: Press a button exactly 3 times. Without memory, the robot either stops too early or never stops.
• Tracking invisible progress: Wipe a bowl 2 full rounds. Round one and round two look identical through the camera.
• Search with partial observability: Lift two overturned cups to find a hidden block. After replacing the first cup, the scene looks exactly as it did at the start — no visual cue indicating which cup was already checked.

These tasks are fundamentally non-Markovian: the correct action depends on the history of interactions, not just the current observation.

Why Visual Memory Falls Short

Recent papers like MemoryVLA and MEM have added visual memory to VLAs by caching past image frames. But this approach has two critical weaknesses:

Computational cost. Every past image frame adds hundreds of tokens. Storing K=5 frames increases inference latency by 39ms (60.7ms → 99.8ms); K=16 frames by 129ms. For real-time robot control, this is significant.

Visual ambiguity. When pressing a button with only a few millimeters of travel, the image is nearly identical before and after each press. Visual memory cannot distinguish "pressed once" from "pressed twice" because the pixel difference is negligible.

In FM-VLA's experiments, π-MEM (the strongest visual memory baseline) achieves only 33.3% success on the button-pressing task — close to random. FM-VLA achieves 72.2% on the same task.

The Core Insight: Force Is Natural Memory

When a robot touches an object, the F/T sensor produces a clear spike — far more informative and consistent than any pixel change. Each contact event leaves a "fingerprint" in the force time series, independent of lighting, camera angle, or object color.

FM-VLA exploits this: instead of remembering images, the model remembers wrist force/torque history from a 6-axis F/T sensor. The history is compressed into a small set of Force Memory Tokens, then injected into the VLA action expert to guide decision-making.

Architecture Overview

FM-VLA builds on π0.5 (Physical Intelligence) — a VLA combining a VLM backbone (PaliGemma + SigLIP) with a flow-matching action expert. The FM-VLA additions consist of three parts:

1. Wrench Signal Preprocessing

Raw F/T sensor readings at 100Hz are noisy. Before encoding, the signal is processed with:

• Causal EMA smoothing: f̃_τ = α·f_τ + (1-α)·f̃_{τ-1} — removes high-frequency noise while preserving contact onset and peak events.
• Randomized noise pre-padding: During training only, prepend each history with a random-length prefix of low-amplitude Gaussian noise. This prevents the model from learning a shortcut based on sequence length (e.g., "short sequence = early in episode"). Disabled at inference.

2. Force-VAE Encoder (Perceiver-IO)

The central component of FM-VLA. A Variational Autoencoder (VAE) is pretrained to compress an entire episode's force history into K = 8 latent tokens (each a d_z-dimensional vector).

The VAE uses a Perceiver-IO architecture, enabling it to handle variable-length inputs (a 10-second episode or a 60-second episode) into the same fixed number of output tokens. The process:

1. Normalize force readings per timestep using per-channel quantile statistics
2. Project through input MLP + add Fourier positional encoding
3. Encoder cross-attention: 8 learnable query tokens attend to the full wrench sequence → posterior parameters (μ_k, log σ²_k)
4. Sample: z_k = μ_k + σ_k ⊙ ε_k, ε~N(0,I)
5. Decoder reverses the process to reconstruct the force sequence (training only)

Training objective: masked ELBO with free-bits KL regularization to prevent posterior collapse:

# Force-VAE loss
L_recon = masked_mse(f_hat, f, mask)  # only over valid (non-padded) frames
L_kl = sum(max(KL_per_dim, lambda_free_bits)) / (K * d_z)
L_vae = L_recon + beta * L_kl

Why VAE over GRU or Q-Former?

• GRU suffers from vanishing gradients over long 100Hz sequences (a 60s episode = 6000 timesteps)
• Q-Former overfits to instantaneous force peaks instead of learning holistic temporal structure
• VAE reconstruction objective forces the latent space to encode macroscopic patterns (force magnitudes, contact counts, onset timings) in just a few tokens

Ablation results confirm this: FM-VLA(GRU) = 33.3%, FM-VLA(Q-Former) = 57.4%, FM-VLA(VAE) = 83.3%.

3. Short State History Token

Force memory alone is insufficient. Before contact, the robot needs to know "where are the arms right now" to move accurately. FM-VLA adds a state history token from a short window of the most recent joint positions (covering ~1 second).

The joint-state window S_t ∈ R^{W×d_s} (W frames, d_s = 16 for bimanual 7-DoF + 2 grippers) is flattened and projected via a single linear layer (no pretraining required) into one token.

4. Memory Token Injection

At inference, the action expert sequence layout is:

[noisy action tokens] || [Z_f^(1), ..., Z_f^(8)] || [z_s]

Force and state tokens are appended after noisy action tokens, preserving the base model's RoPE positional encoding — crucial for not disrupting π0.5's pretrained action generation.

Total additional tokens: 9 (8 force + 1 state). Compare this to π-MEM's hundreds of visual tokens.

Two-Stage Training Pipeline

Stage 1: Force-VAE Pretraining (Task-Agnostic)

# Train VAE on wrench histories across ALL tasks
# No labels needed — only force reconstruction
python train_force_vae.py \
  --data_root ./data/wrench_histories \
  --tasks task1 task2 task3 \
  --n_latents 8 \
  --latent_dim 64 \
  --beta 0.1 \
  --free_bits_lambda 0.5 \
  --epochs 100 \
  --batch_size 32

Important: Train jointly across all tasks using inverse-frequency sampling — tasks with fewer demonstrations are upsampled so small datasets are not dominated by larger ones. Task 1 (200 demos) and Task 3 (200 demos) get higher sample weights than Task 2 (350 demos).

Stage 2: VLA Fine-tuning

With the Force-VAE trained, freeze the encoder and fine-tune the VLA:

python finetune_vla.py \
  --base_model pi0.5 \
  --force_vae_ckpt ./checkpoints/force_vae_best.pt \
  --data_root ./data/demonstrations \
  --n_force_tokens 8 \
  --state_window_size 15 \
  --lr 1e-4 \
  --epochs 50 \
  --tasks task1 task2 task3

What gets trained in Stage 2:

• VLM encoder (PaliGemma + SigLIP) — unfrozen
• Flow-matching action expert — unfrozen
• Short state projector Proj_ψ — unfrozen (zero-initialized)
• Wrench latent projector W_f — unfrozen (zero-initialized)
• Force-VAE encoder — frozen (use only posterior mean μ_f, no reparameterization)

The objective remains π0.5's rectified flow matching loss, now conditioned on Z_f and z_s:

# Flow matching loss with force + state conditioning
v_pred = action_expert(a_k, timestep=k, context=c_t,
                        force_tokens=Z_f, state_token=z_s)
loss = mse(v_pred, epsilon - a_0)

Data Collection: VR Teleoperation on AgiBot G1

FM-VLA is evaluated on the AgiBot G1 — a bimanual humanoid robot with two 7-DoF arms and two 1-DoF grippers. Each wrist is equipped with a 6-axis wrench sensor measuring 3-axis force + 3-axis torque at 100Hz.

Demonstrations were collected via a VR-based teleoperation interface:

• Task 1 (Find Block Under Cups): 200 demos
• Task 2 (Press Button N times): 350 demos
• Task 3 (Wipe Bowl N times): 200 demos

If replicating on another robot, you need:

• 6-axis F/T sensor on the end-effector or wrist
• Sampling rate ≥ 50Hz (100Hz preferred)
• Time-synchronized F/T, cameras, and joint states

What the Force Signals Look Like

Each task has a distinct force "signature":

• Task 1 (block search): Two clear spikes corresponding to lifting two cups
• Task 2 (button press): N sharp, short impulses (each press is a few milliseconds)
• Task 3 (bowl wipe): N sinusoidal cycles corresponding to N wiping passes

The VAE learns to encode these patterns into 8 latent tokens without any task labels. This is unsupervised temporal structure learning — the model discovers contact events purely from the reconstruction objective.

Results

Main Comparison

Key observations:

• TA-VLA underperforms the base model: Short force windows introduce noise into action prediction without providing useful memory. Wrong conditioning is worse than no conditioning.
• Visual memory completely fails on Buttons (33.3%): As predicted — button presses leave no visual trace.
• FM-VLA achieves 100% on Cups: This hybrid task requires both spatial reasoning (which cup was already checked) and contact feedback — force + state history handles both dimensions perfectly.

Task Demos

Inference Efficiency

9 tokens vs. hundreds of visual tokens — that is why FM-VLA's overhead is negligible. Running at 15+ Hz on an RTX 4090 is completely feasible.

Ablation Studies: Unpacking Each Design Decision

What to remember?

• Force only: Without state history, the policy lacks short-term spatial awareness before contact — arms move erratically during approach
• State only: Perfect for the spatially-driven Cups task, but completely fails at counting contact events
• Combined: Best of both worlds

How many tokens: K = ?

Tested at {4, 8, 16, 32}:

• K=4: Information bottleneck — 77.8% on Wipe
• K=8: Optimal — peak success rate
• K=16: Performance drops — exceeds the pretrained action expert's effective context window (π0.5 was pretrained seeing at most ~50 tokens)
• K=32: Disrupts coherent action generation

This is a crucial practical lesson: more tokens is not always better. You must respect the constraints of the base model you are fine-tuning.

Installation and Running FM-VLA

Code is being prepared for release at github.com/qft-333/FM-VLA. While waiting, set up a compatible environment:

# Clone repo (once code is released)
git clone https://github.com/qft-333/FM-VLA.git
cd FM-VLA

# Create environment (compatible with pi0.5 + wrench processing)
conda create -n fmvla python=3.10
conda activate fmvla
pip install torch==2.4.0 torchvision
pip install transformers accelerate
pip install einops perceiver-pytorch

# For AgiBot G1 with ROS 2
sudo apt install ros-jazzy-ros-base
pip install rclpy sensor_msgs_py

Required Data Format

# Each demo episode should be stored as:
{
  "images": {
    "head": [...],         # list of H×W×3 arrays
    "left_wrist": [...],
    "right_wrist": [...]
  },
  "wrench": {
    "left": [...],    # T×6 array [Fx, Fy, Fz, Tx, Ty, Tz] at 100Hz
    "right": [...]
  },
  "joint_states": [...],   # T×16 array [q_left×7, gripper_left, q_right×7, gripper_right]
  "actions": [...],        # T×16 array
  "language": "press the button 2 times"
}

Inference with a Trained Model

import torch
from fm_vla import FMVLA, WrenchHistory

# Load model
model = FMVLA.from_pretrained("./checkpoints/fmvla_final")
model.eval()

# Initialize wrench history buffer
wrench_buf = WrenchHistory(ema_alpha=0.3)

# Inference loop
for obs in robot.stream():
    # Update wrench history with new reading
    wrench_buf.update(obs["wrench"])  # 6D wrench reading
    
    # Run FM-VLA
    with torch.no_grad():
        action = model.predict(
            images=obs["images"],
            language=task_instruction,
            wrench_history=wrench_buf.get_history(),
            joint_states=obs["joint_states"]
        )
    
    robot.execute(action)

Unlike force-conditioned VLA approaches like FD-VLA that use force for immediate correction, FM-VLA uses force primarily as an episodic memory — to remember what happened, not just what is happening now.

Limitations and Future Directions

Current limitations:

1. Fixed 8-token VAE bottleneck — for very long-horizon tasks (>100 contact events), 8 tokens may not capture all relevant history
2. VAE pretrained on the lab's own demonstration data — pretraining on a large-scale, diverse robot force dataset could improve cross-robot generalization

Promising extensions:

• Hierarchical compression: Two-level VAE — low level captures raw contact, high level captures semantic structure (e.g., "button pressed twice")
• Cross-robot transfer: Contact patterns for similar tasks are consistent across robots — sharing Force-VAE weights may be feasible
• Tactile sensor fusion: Combine 6-axis wrist F/T with tactile skin readings from the gripper

To understand the π0.5 base model that FM-VLA builds on, see the Pelican-VLA π0.5 fine-tuning tutorial.

Conclusion

FM-VLA addresses a genuinely underexplored gap in the VLA literature: non-Markovian contact-rich manipulation — tasks that require remembering interaction history across many steps. The core insight is simple but powerful: force sensors generate natural, unambiguous memory where cameras cannot.

The two-stage design (Force-VAE pretraining → VLA fine-tuning) allows the encoder to learn strong representations without task-specific labels, while adding only 9 tokens and 3ms overhead to the base model.

As VLA-based manipulation continues advancing toward more complex bimanual tasks, FM-VLA establishes a blueprint for giving VLAs sensory memory — not just eyes, but also hands that remember.

Related Posts

• FineVLA: Fine-tuning VLA for Dual-Arm Manipulation
• Dexora: Bimanual Dexterous Manipulation with VLA — Teaching Robots Two-Handed Skill
• AXIS: Browser Teleoperation + π0.5 Fine-Tuning for Manipulation Data Collection