FM-VLA: Force Memory Teaches VLA Contact-Rich >80%

Imagine telling a robot to press a button exactly three times. After each press, the button looks identical. Nothing visually changes. How does the robot know whether it has pressed once, twice, or three times already?

This is the non-Markovian problem in robotics: tasks where the current visual state is insufficient to decide the next action. And it is a fundamental blind spot in nearly every modern Vision-Language-Action (VLA) model.

FM-VLA — from researchers at Tsinghua University, Microsoft Research, Fudan University, and USTC — attacks this problem with an elegant answer: instead of looking back at past images (expensive, and blind to visually-ambiguous events), feel the history through a force/torque sensor.

Paper: FM-VLA: Force-based Memory for Vision-Language-Action Models in Contact-Rich Manipulation — Li et al., 2026.

The Problem: VLAs Have No Memory of Physical Events

Modern VLAs like π₀ (pi-zero) operate Markovianly — every decision depends only on the current frame. This works fine for most manipulation tasks: pick an object, place it in a bin, sort items. But when the task requires counting contact events or tracking a sequence of physical interactions, the model is completely blind.

Some groups addressed this with visual memory — caching past image frames and feeding them back to the model. The issues:

• Massive compute overhead — each additional frame adds 39–129ms to inference latency.
• Blind to visually-ambiguous events — pressing a button multiple times creates nearly zero visual change. The camera cannot distinguish "pressed once" from "pressed three times" if the button springs back.

FM-VLA asks the inverse question: What always changes during physical contact, even when the camera sees nothing different? The answer: the force/torque sensor.

Architecture: Three Synergistic Streams

FM-VLA is built on π₀.₅ — a PaliGemma VLM backbone paired with a flow-matching action expert. The additions are two proprioceptive memory streams injected into the action expert.

Stream 1: Force History Collection

The AgiBot G1 bimanual humanoid robot (2×7-DoF arms with parallel grippers) carries two 6-axis force/torque sensors at the right wrist, sampling at 30Hz. Each frame yields 6 values: forces along three axes (Fx, Fy, Fz) and torques along three axes (Mx, My, Mz).

Before entering the encoder, the raw signal is filtered with Exponential Moving Average (EMA) at α=0.3 to remove mechanical noise, yielding a clean time series that faithfully records every contact event during the episode.

Stream 2: Force Memory Encoder (Frozen VAE)

This is the core contribution — a Perceiver-IO Variational Autoencoder pretrained specifically to compress force time series into compact representations.

VAE architecture:

• Input: Quantile-normalized wrench frames → projected to 384-dim with Fourier positional encoding
• Encoder: 2 cross-attention blocks + 10 self-attention blocks with K=8 learnable latent tokens
• Output: 8 tokens of 96-dim, compressed to the action expert's hidden width

Critically, the VAE is frozen after pretraining. Only projector layers are fine-tuned alongside the VLA. This prevents catastrophic forgetting of the force representation.

Why K=8? The ablation found K=4 creates an information bottleneck — insufficient to encode a full manipulation episode. K=16+ exceeds the pretraining distribution. K=8 is the sweet spot.

Stream 3: Short-window State History

Alongside the long-horizon force memory, FM-VLA adds a short-term spatial stream: the last 10 joint-state frames (0.9 seconds at 30Hz) are flattened and projected into a single token via a zero-initialized linear layer.

This token provides pre-contact spatial awareness — the robot knows where its arm is in space before reading the force event history.

Injection into the Action Expert

The action expert input sequence is extended to:

[noisy actions (30 tokens) || force memory (8 tokens) || state window (1 token)]

Only 9 additional tokens — explaining why the inference overhead is just +3.3ms (total 64ms vs 60.7ms baseline).

Training Pipeline: Two Stages

Stage 1: Force-VAE Pretraining

Objective: Masked ELBO on wrench reconstruction
Steps: 100k
Batch size: 512
KL weight: β = 1×10⁻³
Free-bits floor: λ = 0.5 nats per dimension
Task sampling: Inverse-frequency (avoids overfitting easy tasks)

This stage requires no action labels — only wrench sequences from teleoperation episodes. The VAE learns to summarize all contact events into a compact, reconstructable form.

Key trick: Randomized noise padding. During training, the start of each wrench sequence is padded with Gaussian noise (up to 10 seconds). This prevents the model from exploiting sequence length to infer task progress — forcing it to learn contact content, not absolute timing.

Stage 2: VLA Fine-tuning

Base: π₀.₅ (PaliGemma VLM + flow-matching action expert)
Steps: 50k
Batch size: 32
Peak LR: 5×10⁻⁵
Warmup: 1k steps
Hardware: 8×A100 (40GB) for training, RTX 4090 for inference

During this stage:

• VAE encoder is completely frozen (eval mode only)
• Only the posterior mean μf is used — no reparameterization noise at inference
• VLM, action expert, state projector, and force projector are all trained jointly

Dataset collected via VR teleoperation:

Evaluation Results: 83.3% vs 27.8%

The core results, compared against all baselines:

Breaking down the results:

Cups task (find hidden block): FM-VLA achieves 100% — the robot knows exactly which cup was already lifted (via force spike) and stops searching immediately. Visual-memory baselines cannot distinguish which cup was already checked.

Buttons task (press exactly N times): The hardest task. π₀.₅ fails catastrophically at 11.1% — no counting ability. FM-VLA reaches 72.2% because each button press creates a distinctive force spike in memory, enabling reliable counting.

Wipe task (wipe bowl N rounds): π₀.₅ completely fails at 0% — without memory, the robot cannot know how many rounds it has completed. FM-VLA achieves 77.8% by accumulating the cyclic force pattern in memory tokens.

Inference Latency Breakdown

FM-VLA matches the latency of text-summary approaches while dramatically outperforming them. At 64ms total, it runs comfortably within real-time control loops on an RTX 4090.

Ablation Studies: Why VAE Beats GRU and Q-Former

The team tested multiple alternative encoders for the force history:

GRU fails because: hidden states tend to collapse to zero during non-contact periods, losing information about when contact events occurred in long episodes.

Q-Former does better but still lacks: no inductive bias for time series reconstruction, making it harder to preserve precise event timing.

Frozen VAE wins because: pretrained specifically to reconstruct wrench sequences, it has strong inductive bias that maps contact events to structured latent positions. Freezing it prevents the VLA fine-tuning from degrading this learned representation.

Modality ablation:

Force-only fails at pre-contact phases (no spatial awareness). State-only fails at contact counting. Together, they provide complementary signals: state history for short-term spatial context, force memory for long-horizon event counting.

Deeper Understanding: Why Force Memory Works Here

A force sensor is a side channel immune to visual ambiguity. When a robot presses a button — even if the button springs back to its original position (visually indistinguishable) — the force spike is permanently recorded in the VAE memory. Each contact event leaves a structured imprint in the latent space.

More importantly: force sensors respond instantaneously — no blur, no occlusion, no lighting dependency. In industrial or domestic environments where camera conditions are uncontrolled, this is a major advantage.

Force data is also privacy-respecting: in homes or hospitals where cameras may be restricted, a force sensor records only a stream of numbers meaningless to humans but informative to the robot.

Limitations and Future Directions

FM-VLA in its current form has known limitations worth keeping in mind:

1. Fixed K=8 tokens may be insufficient for very long contact-rich episodes (>30 seconds). The authors propose expanding pretraining with longer force trajectories.

2. Single right-wrist sensor only — complex bimanual contact tasks require dual force sensing, not yet fully exploited.

3. Code not yet released — as of August 2026, the GitHub repo qft-333/FM-VLA is pending code release. Watch the repo for updates.

4. Hardware dependency — requires a 6-axis F/T sensor at the wrist. Not all robot platforms support this out of the box.

How to Implement FM-VLA When Code Drops

Based on the published architecture, here is the workflow you will need to prepare:

Step 1: Collect teleoperation data

# Each task needs 200-350 demos with synchronized F/T sensor logging
# Robot requirements: 6-axis F/T sensor at wrist, 30Hz sampling
# Data format: (image, language, joint_state, wrench, action) tuples per timestep

Step 2: Pretrain the Force-VAE

# Train VAE on wrench sequences only — no action labels needed
# 100k steps, batch 512
# With gradient accumulation, 2-4x A100 is feasible

python train_force_vae.py \
  --data_dir ./data/wrench_sequences \
  --latent_tokens 8 \
  --steps 100000 \
  --batch_size 512 \
  --kl_beta 1e-3 \
  --free_bits 0.5

Step 3: Fine-tune VLA with Force Memory

# Load pretrained VAE (freeze it)
# Fine-tune pi0.5 + state projector + force projector end-to-end

python train_fm_vla.py \
  --base_model pi05 \
  --force_vae_path ./checkpoints/force_vae.pt \
  --freeze_vae True \
  --data_dir ./data/task_demos \
  --steps 50000 \
  --lr 5e-5 \
  --warmup_steps 1000

Step 4: Deploy on real robot

# Runtime loop: encode accumulated force history → tokens
# Append to action expert sequence alongside state history
# Total inference: ~64ms on RTX 4090

robot.deploy(
  model="fm_vla",
  force_sensor=wrist_ft_sensor,
  language_instruction="Press the button 3 times"
)

Why This Matters Beyond the Paper

Most VLA research today targets generalization — making robots handle more diverse tasks. FM-VLA solves a different problem: temporal reasoning — making robots track sequences of physical events over time.

This is an essential step for robotics entering real-world deployment:

• Manufacturing: tighten a bolt the correct number of turns
• Healthcare: perform physiotherapy exercises with exact repetition counts
• Quality inspection: execute test button protocols in precise order

The architecture pattern is generalizable beyond force sensors. The core insight — pretrain a frozen encoder for your sensor modality, then inject its latent representations into an existing VLA backbone — applies equally to tactile sensors, audio, thermal cameras, or any sensor that captures information invisible to RGB cameras.

If you are building a VLA pipeline for contact-rich tasks today, FM-VLA provides a clear and battle-tested blueprint: do not fight visual ambiguity with more vision. Complement it with a dedicated physical sensor and a pretrained encoder that understands that sensor's language.

Related Posts

• Force Control in Robot Manipulation — From PD to Impedance Control
• Contact-Rich Manipulation: Curriculum RL for Complex Tasks
• UniTac-VLA: Tactile Chain-of-Thought for Contact Manipulation