Guided Action Flow for SmolVLA

Guided Action Flow, or GAF, is a practical idea for robot manipulation: keep the base Vision-Language-Action policy frozen, then add a small action-chunk critic at inference time. The critic does not replace the policy, does not rerank a finite set of completed action samples, and does not backpropagate through the full VLA. It only provides gradients with respect to the action chunk so the flow sampler is nudged toward higher-value actions.

This article is based on the paper Guided Action Flow: Q-Guided Inference for Flow-Matching Vision-Language-Action Policies, arXiv 2607.02092, and the official repository ylhaichen/guided-action-flow. The goal is to make the method followable for a beginner: the paper idea, architecture, installation, rollout collection, critic training, Q-guided inference, and how to interpret the LIBERO results. If you are new to VLA policies, start with VLA Models in robotics and then read a LeRobot hands-on guide from the AI robotics series.

Why GAF Exists

Modern VLA policies such as OpenVLA, pi0, GR00T, and SmolVLA are often trained with imitation learning on demonstrations. That is powerful when the deployment task is close to the training data. But when a policy fails in a subset of states, improving it usually means collecting more demonstrations, fine-tuning, or running reinforcement learning. For larger models, policy fine-tuning is expensive, can overfit, and changes behavior that may already have been validated.

GAF asks a narrower question: if the frozen policy already has useful competence, can a small critic improve the action sampler during inference? This question is natural for SmolVLA because SmolVLA is a flow-matching policy. It does not output the final action chunk in a single step. It iteratively transports noise into a clean action chunk. That iterative process gives the critic a place to intervene.

Intuitively, imagine SmolVLA generating a 50-step action chunk for a LIBERO task. At every denoising step, the sampler has an intermediate action x_t and a velocity v_t. If a critic can score which action chunks are more likely to succeed, we can differentiate that score with respect to the estimated clean action a_hat, then slightly adjust the velocity. If the critic is right, the final action can improve. If the critic is wrong or overconfident out of distribution, guidance can create regressions. That is why the paper emphasizes critic ensembles, gradient clipping, and uncertainty gates.

Official Sources

The paper is explicit about its scope. GAF is an early empirical study, not a claim that Q-guided inference has solved VLA adaptation. The strongest evidence is that critic gradients can improve a frozen SmolVLA policy in real LIBERO rollouts, while critic generalization remains the central bottleneck. The repository also describes itself as project glue: it does not vendor LeRobot, SmolVLA, or LIBERO into the main package. Upstream repositories live under third_party/, checkpoints under checkpoints/, and rollout artifacts under runs/.

Architecture Overview

GAF has three main components.

The key constraint is that the policy is not updated. SmolVLA still runs normally: observations pass through the vision-language pathway, the action expert predicts flow velocity, and the sampler integrates from noise toward action. GAF installs a hook inside the denoising loop. The hook receives x_t, v_t, timestep t, observation features, and task features; it estimates the clean action, queries the critic, and returns a guided velocity.

The repository is organized around these boundaries:

src/guided_action_flow/
  benchmarks/      adapters for LIBERO, LIBERO-plus, LIBERO-PRO
  critics/         action-chunk critic, ensemble, checkpoint loading
  guidance/        QGF update rule
  policies/        SmolVLA/LeRobot wrapper and QGF hook
  rewards/         reward and success extraction
  training/        rollout dataset, returns, task features
  evaluation/      rollout and metrics

If you have used diffusion guidance in image generation, the pattern will feel familiar: a base generative model produces a sample, and an external signal steers the sampling trajectory. Robotics makes the problem stricter. The generated sample is not an image; it is an action chunk that must obey action scale, gripper semantics, robot kinematics, and closed-loop execution.

SmolVLA Flow Convention

The easiest technical detail to get wrong is the guidance sign. According to the repository docs, the pinned SmolVLA implementation uses this reverse-time convention:

x_t = t * noise + (1 - t) * action
v_t = noise - action

Inference starts from noise at t = 1 and moves toward action at t = 0:

dt = -1 / num_steps
x_t = x_t + dt * v_t

Therefore the estimated clean action at an intermediate step is:

a_hat = x_t - t * v_t

If you copy a Q-guided flow update from a paper that uses the opposite time convention, you can move the sampler in the wrong direction. For the pinned SmolVLA code path used by GAF, increasing the clean action along the critic gradient +g requires decreasing the velocity:

g = grad_a Q(obs_features, proprio, a_hat)
v_guided = v_t - g / beta

The fuller version uses an ensemble, gradient clipping, and a disagreement gate:

q_mean = mean(Q_k(obs_features, a_hat, task_feature))
g = grad_a q_mean
g = clip_by_norm(g, c)
gate = max(m_min, exp(-alpha * std(Q_k)))
v_guided = v_t - gate * g / beta

Smaller beta means stronger guidance. grad_clip_norm prevents unstable critic gradients from dominating the action sampler. std(Q_k) measures disagreement across ensemble members; when critics disagree, the gate reduces the guidance strength.

What the Critic Learns

The GAF critic does not predict actions. It predicts a scalar value for an action chunk. The basic inputs are policy-side observation features, proprioception when used, the 7D LIBERO action chunk, and an optional task feature. The target is sparse success-to-go from real rollouts of the frozen SmolVLA policy.

Consider an 80-step episode. If success first appears at step 60, chunks starting before that step receive discounted targets, and chunks after success receive targets near 1. If the episode never succeeds, the target is 0. This target is cheap and easy to debug, but it is also blunt. It may not distinguish a near-success trajectory from a completely bad one if both fail before the success checker fires.

The repository emphasizes that train/validation splits must happen at the episode level, not the chunk level. If you randomly split overlapping chunks, adjacent chunks from the same trajectory can leak into both train and validation. The validation loss will look better than the critic actually is. For beginners, this is one of the most important robotics data lessons in the project.

The strongest task-conditioned critic in the paper uses hidden states from the frozen SmolVLA language pathway. The task description is processed by SmolVLA's VLM text path, then non-padding token hidden states are mean-pooled into a task feature. This is stronger than task IDs because task IDs are not portable across LIBERO families. It also avoids training a separate text encoder.

Installation

Linux is strongly preferred because LIBERO depends on MuJoCo and robosuite. The repository notes that the current runs were produced on a laptop RTX 4070 with 8GB VRAM. Official SmolVLA 0.45B LIBERO evaluation and QGF evaluation with K=3 critics both fit on that machine. That makes GAF attractive for smaller labs: you do not need a large cluster to reproduce the first ablations.

Create the Python environment:

conda create -n gaf-libero python=3.12 -y
conda activate gaf-libero
python -m pip install --upgrade pip setuptools wheel
python -m pip install -e ".[dev,torch]"

Bootstrap upstream repositories:

bash scripts/bootstrap_third_party.sh
git clone https://github.com/sylvestf/LIBERO-plus.git third_party/LIBERO-plus
git clone https://github.com/Zxy-MLlab/LIBERO-PRO.git third_party/LIBERO-PRO

Install LeRobot from the pinned checkout:

cd third_party/lerobot
python -m pip install -e ".[smolvla,libero]"
cd ../..

Set up headless MuJoCo:

export MUJOCO_GL=egl
export WANDB_MODE=disabled
export PYTHONPATH="$PWD/src:$PWD/third_party/lerobot/src${PYTHONPATH:+:$PYTHONPATH}"

Create local LIBERO config directories:

mkdir -p .libero_configs/vanilla .libero_configs/plus .libero_configs/pro
export LIBERO_CONFIG_PATH="$PWD/.libero_configs/vanilla"

The .libero_configs/* files contain absolute paths to assets, BDDL files, benchmark root, datasets, and init states. They are machine-specific and should not be committed. If the simulator fails before the policy is called, check LIBERO_CONFIG_PATH, MuJoCo EGL, asset paths, robosuite version, and the installed LIBERO package before tuning the critic.

Download Checkpoints and Run Baseline

GAF uses the official SmolVLA checkpoint for LIBERO:

python scripts/download_models.py \
  --repo-id lerobot/smolvla_libero \
  --local-dir checkpoints/smolvla_libero

Run a one-episode smoke test:

PYTHON_BIN="$CONDA_PREFIX/bin/python" \
POLICY_PATH="$PWD/checkpoints/smolvla_libero" \
OUTPUT_DIR="$PWD/runs/smolvla_libero_smoke" \
TASK=libero_spatial \
TASK_IDS='[0]' \
N_EPISODES=1 \
bash scripts/eval_smolvla_libero.sh

Run the baseline with the general evaluator:

python scripts/eval_policy.py \
  --policy-path checkpoints/smolvla_libero \
  --output-dir runs/baseline_spatial3_ep50_seed3000 \
  --env-type libero \
  --task libero_spatial \
  --task-ids '[3]' \
  --n-episodes 50 \
  --seed 3000 \
  --device cuda \
  --max-videos 0

Do not skip the baseline. If your baseline is much lower than the repository report, do not tune QGF yet. Check checkpoint paths, action normalization, camera keys, task IDs, seed handling, maximum episode length, LeRobot version, and LIBERO version. GAF is a small guidance layer; it will not repair a broken evaluation pipeline.

Collect Rollouts for the Critic

The critic must learn from real rollouts of the frozen SmolVLA policy, not dummy actions. For a single-task trial:

python scripts/collect_rollouts.py \
  --policy-path checkpoints/smolvla_libero \
  --output-dir runs/qgf_single_task_spatial3_train50 \
  --env-type libero \
  --task libero_spatial \
  --task-ids '[3]' \
  --n-episodes 50 \
  --seed 2000 \
  --device cuda

The output under runs/ contains episode trajectories, action sequences, success flags, and metadata. The dataset builder converts these trajectories into overlapping action chunks. With horizon 50, an 80-step episode can produce many chunks starting at different steps. The observation feature is taken from the start of the chunk, the action input is the following action sequence, and the target is success-to-go.

Before training, check:

Train a Single-Task Critic

The basic critic training command is:

python scripts/train_critic.py \
  --data-dir runs/qgf_single_task_spatial3_train50 \
  --output-dir runs/qgf_single_task_spatial3_critic_train50 \
  --action-horizon 50 \
  --hidden-dim 512 \
  --depth 3 \
  --epochs 20 \
  --seed 0 \
  --device cuda

The paper and README also describe stronger variants: a K=3 critic ensemble, hidden dimension 768, depth 4, 30 epochs, and task-description features from SmolVLA VLM hidden states. For a first run, however, the single-task critic is the right debugging step. It verifies the entire path: rollout recording, success-to-go targets, action chunk extraction, checkpoint saving, and autograd through the action input.

A common runtime issue is that the outer evaluator may call policy.select_action() under torch.inference_mode(), while GAF needs autograd.grad for the critic path. The repository handles this by using torch.inference_mode(False) and torch.enable_grad() only around critic differentiation. The denoiser output is detached before the critic query, so gradients do not flow into SmolVLA.

Enable Q-Guided Inference

After training a critic:

python scripts/eval_policy.py \
  --policy-path checkpoints/smolvla_libero \
  --output-dir runs/qgf_single_task_spatial3_eval50_qgf_beta2_seed3000 \
  --env-type libero \
  --task libero_spatial \
  --task-ids '[3]' \
  --n-episodes 50 \
  --seed 3000 \
  --device cuda \
  --max-videos 0 \
  --critic-path runs/qgf_single_task_spatial3_critic_train50/critic.pt \
  --qgf-beta 2 \
  --qgf-grad-clip-norm 1.0

For ensemble guidance with an adaptive gate:

python scripts/eval_policy.py \
  --policy-path checkpoints/smolvla_libero \
  --output-dir runs/qgf_vlm_hidden_val_spatial_5_7_8_beta2_gate20_seed8000 \
  --env-type libero \
  --task libero_spatial \
  --task-ids '[5, 7, 8]' \
  --n-episodes 10 \
  --seed 8000 \
  --device cuda \
  --max-videos 0 \
  --critic-paths \
    runs/qgf_multitask_spatial0to4_critic_vlm_hidden_seed0/critic.pt \
    runs/qgf_multitask_spatial0to4_critic_vlm_hidden_seed1/critic.pt \
    runs/qgf_multitask_spatial0to4_critic_vlm_hidden_seed2/critic.pt \
  --qgf-beta 2 \
  --qgf-grad-clip-norm 1.0 \
  --qgf-uncertainty-scale 20 \
  --qgf-min-gate 0.1

When reading logs, look beyond success rate. Inspect q_guidance_norm_mean, the number of guided denoising steps, gains and regressions per episode, and critic disagreement. A setting can match baseline success while changing which episodes succeed. That still matters because it proves the guidance path is active; the critic is just not reliable enough yet.

Results

The main GAF numbers are:

The repository also reports baseline anchors:

The correct interpretation is cautious. Single-task guidance shows a strong signal. Multi-family validation improves clearly. But the locked held-out gain is modest. This is not evidence that "adding a critic always wins." The paper shows that critic generalization is the main problem. A spatial-only critic may not transfer to object tasks. Overly strong guidance can create regressions. A weak uncertainty gate may fail to protect the sampler outside the critic's training distribution.

When to Use GAF

GAF makes sense when you already have a frozen flow-matching policy with nontrivial task competence, a simulator or robot setup for collecting success/failure rollouts, and a narrow task family you want to improve without full policy fine-tuning. It is especially attractive for SmolVLA and LeRobot users because the cost is much lower than RL fine-tuning a large VLA.

Do not use GAF as a patch for a policy that has not learned the task at all. If baseline success is near zero, the critic will not have enough successful rollouts to learn from. If the observation pipeline is wrong, the critic will learn noise. If the deployment task is far from LIBERO, you need stronger OOD detection and real deployment data.

Minimum checklist:

[ ] SmolVLA baseline runs correctly on the target task
[ ] Rollout dataset contains both successes and failures
[ ] Critic inputs are available at inference time
[ ] Train/validation split is episode-level
[ ] QGF sign is verified against the actual sampler
[ ] Beta, clip norm, and gate are chosen on validation
[ ] Held-out test is opened only after hyperparameters are frozen

Relation to Other VLA Directions

GAF sits between imitation learning and reinforcement learning. It does not require additional expert demonstrations like supervised fine-tuning. It also does not update the policy with RL gradients. The critic learns from rollouts, then intervenes only while actions are sampled. Compared with fine-tuning VLA on LIBERO with Embodied-R1.5, Embodied-R1.5 changes the policy checkpoint, while GAF keeps SmolVLA frozen. Compared with SLIM-0.5B on LIBERO, GAF does not propose a new backbone; it wraps a guidance layer around an existing sampler.

The method is also related to value-guided generation in diffusion and flow models. But robot actions are not images. They are constrained by kinematics, gripper state, action normalization, control frequency, and closed-loop execution. That is why the paper is careful: critic gradients can help, but they must be clipped, gated, and evaluated with real rollouts.

Conclusion

Guided Action Flow is interesting because the workflow is compact: freeze SmolVLA, collect LIBERO rollouts, train an action-chunk critic, then use critic gradients to guide the reverse-time flow sampler. The core idea is simple: instead of making the policy learn again, use a value signal to steer action generation during inference.

The strength of the method is that it is modular, affordable, and easy to disable. Turning off the critic returns you to the SmolVLA baseline. The weakness is critic generalization. The single-task improvement from 68.0% to 82.0% is meaningful, but the locked held-out test moves only from 65.0% to 67.5%. Deployment should therefore be experimental and disciplined: validate carefully, use ensemble gating, avoid tuning on test tasks, and do not expect a small critic to solve every distribution shift.

For beginners, the main lesson is bigger than v_guided = v_t - g / beta. GAF is a clean pattern for robot learning adaptation: frozen base policy, small critic, rollout-derived data, simple targets, and separated validation/test evaluation. That pattern is worth learning even if future versions replace the critic target or improve the OOD gate.

Related Posts

• VLA Models in robotics
• LeRobot hands-on guide
• SLIM-0.5B: Lightweight VLA on LIBERO