DIRECT: Routing robot compute

If you build robot manipulation systems with VLMs or VLAs, the practical question is not only "which model is the smartest?" A more useful question is: "when is it worth paying extra latency, tokens, and FLOPs for the smarter planner?" DIRECT: When and Where Should You Allocate Test-Time Compute in Embodied Planners?, by Jadelynn Dao, Milan Ganai, and collaborators, studies exactly that question for embodied planning. Instead of always calling the strongest planner, DIRECT trains a lightweight router that reads the scene image and instruction, then chooses the planner with the best quality-cost trade-off for that specific task.

The key idea is that test-time compute is not one uniform knob. In robotics, spending more compute at inference can mean deeper chain-of-thought reasoning, a larger model size, or longer memory history. These axes buy different capabilities. Reasoning helps when the task contains hidden semantic, physical, or spatial constraints. Larger models broaden the set of skills a planner can command reliably. Memory helps when the robot must use information that is no longer visible in the current frame. DIRECT turns this observation into a router that runs before the planner. Its overhead is on the order of tens of milliseconds, but it can save seconds or tens of seconds of planning time.

The primary sources are the official project page at https://jadee-dao.github.io/direct/ and arXiv 2606.12402, posted on June 10, 2026. The project page includes the video overview, Franka/DROID hardware demos, framework diagrams, and headline results. The code button currently says Code (coming soon), so there is no official public GitHub repository to clone at the time of writing. For that reason, the installation section below describes a minimal reproduction path: build a DIRECT-style router using Python, image/text embeddings, a quality-cost matrix, and a fixed pool of planners already available in your lab. Once the official repo is released, the same mental model should make the code much easier to read.

Paper idea

Many modern robot stacks use a two-level hierarchy. The upper level is a VLM planner: it takes a scene image and a natural-language instruction, then decomposes the task into primitive sub-skills such as pick banana, place in white bin, wipe table, or close drawer. The lower level is a language-conditioned policy or VLA policy that executes each sub-skill on the robot. This hierarchy is attractive because the planner handles semantic task understanding, while the low-level policy handles continuous control.

The bottleneck is that the planner is often called repeatedly. In a long-horizon task, the robot completes one step, the scene changes, a transition detector triggers replanning, and the planner is called again. If the planner is a slow thinking model, every call can take many seconds. In a research demo, that latency is annoying. In a deployed robot, it reduces throughput, increases timeout risk, and raises API or GPU cost.

DIRECT starts from a simple observation: not every step needs the same level of reasoning. If the instruction is "put the red cup in the bin" and the object is unambiguous, a cheap planner may be enough. If the instruction is "place the fruits from heaviest to lightest," the robot must reason about hidden world knowledge and ordering; a thinking model may be worth the cost. If the target was mentioned earlier in the episode but is no longer visible, a memory planner may be necessary. DIRECT learns to predict these demands from the multimodal task context before it calls the expensive planner.

A practical way to read DIRECT is as a dispatcher:

RGB scene / multi-view stack + language instruction
        |
        v
Frozen vision encoder + frozen text encoder
        |
        v
Lightweight router
        |
        +--> cheap planner: no-thinking / small / no-memory
        +--> expensive planner: thinking / large / memory
        +--> intermediate planner if the pool has more models
        |
        v
Primitive skill plan
        |
        v
Low-level VLA policy executes on robot

The router does not control the robot directly. It only chooses the planner. The selected planner still emits a skill sequence, and the low-level policy still executes that sequence. This makes DIRECT relatively easy to integrate into existing robot systems: it wraps the planner layer rather than replacing the VLA controller.

DIRECT architecture

The paper defines a fixed planner pool M = {m1, ..., mK}. Each planner receives a task x = (I, l), where I is a scene image or a multi-view stack, and l is the instruction. The planner outputs a sequence of primitive sub-skills. For every task and every planner, the authors measure two quantities:

The router training data therefore consists of two matrices, Q and C, over the task-planner grid. In simulation, quality can be measured by benchmark rollouts. On hardware, exhaustively running every planner on every real scene is too expensive, so the paper synthesizes physical-style training data: sample scenes disjoint from evaluation, prompt a large VLM to propose candidate instructions and reference skill decompositions, run each planner to record its sequence and latency, then use an LLM judge to score quality against the reference.

The router consumes a task embedding phi(x). The paper uses a frozen SigLIP-family vision encoder for the image and a frozen BGE-M3 text encoder for the instruction, then concatenates the two embeddings. The authors sweep several lightweight router architectures: linear models, KNN, pairwise-preference KNN, k-means, one-versus-rest classifiers, and two-layer MLPs. Because one embedding pass plus router inference costs roughly 20-50 ms, the overhead is tiny compared with VLM planner calls that usually take more than one second, and especially tiny compared with thinking planners that can take tens of seconds.

The routing objective is not simply "pick the highest quality planner." If it were, the router would always choose the strongest model. DIRECT instead uses a utility function U(q, c) that balances quality and cost. A beginner-friendly version is:

utility = predicted_quality - lambda * predicted_cost
chosen_planner = argmax_k utility(task, planner_k)

The paper reports that regression heads are generally strong because they preserve richer quality and cost signals than a hard class label. At deployment time, the router observes a fresh task, computes the embedding, predicts a planner index, and invokes that planner. For multi-stage tasks, the router can be called again after each completed step because the next scene may have a different difficulty profile.

Three axes of test-time compute

Chain-of-thought: reasoning is not always worth it

On VLABench, the authors compare no-thinking and thinking planner pairs. Thinking models can handle subtle semantic, physical, and spatial constraints, but they generate more tokens and add substantial latency. One memorable result: for Qwen3-VL 8B Instruct versus Qwen3-VL 8B Thinking, the Instruct model matches or beats Thinking on about 44% of tasks, while using less than 2% of the latency. The project page summarizes this as roughly 63x faster in those cases.

This does not mean "never use reasoning." It means reasoning should be allocated. For a clear task, thinking is wasteful. For ambiguity, ordering, hidden constraints, or world knowledge, thinking may prevent failure. DIRECT learns that boundary from the scene and instruction.

Model size: larger does not create a smooth curve

The paper also evaluates Qwen3-VL Instruct variants from 2B to 235B on VLABench. The result is not monotonic: score and latency do not follow a clean scaling curve. Some smaller models generate more verbose outputs and can run slower than larger models. Skill-level analysis suggests that model size mainly expands the breadth of skills that a planner can command reliably. A larger model helps when the task needs a skill the smaller model tends to mis-command, such as fold or close in the hardware validation, but it is unnecessary for obvious put/place steps.

This matters for deployment teams. If your robot always calls a 70B or 235B VLM for every instruction, you are probably overspending on many easy steps. A small router can keep the large model as a specialist and call it only when the small planner is likely to lack the needed skill coverage.

Memory: pay recall overhead only when recall matters

Long-horizon tasks often require memory, but memory can be represented in different ways. The paper discusses FrameSamp and TokenDrop for reducing visual tokens, SimpleSG and GroundSG for summarizing history as language subgoals, and MemER for recalling previous keyframes. On RoboMME, no memory architecture dominates every difficulty tier. For easier tasks with short recall, FrameSamp can outperform MemER with roughly an order of magnitude fewer FLOPs. For harder tasks requiring distant history, MemER and GroundSG become stronger.

The practical message is: do not stuff the entire episode history into every prompt just because the context window allows it. In robotics, long history is compute, and it can also distract the planner when the current frame already contains enough information. DIRECT treats memory as a conditional resource.

Minimal reproduction setup

Because the official repo is not public yet, the best starting point is an independent skeleton. You need four pieces: a task dataset, a planner pool, embedding models, and a router model. For beginners, do this offline first. You do not need a real robot to learn the workflow.

conda create -n direct-router python=3.11 -y
conda activate direct-router
pip install torch torchvision transformers sentence-transformers scikit-learn pandas numpy pillow tqdm

A minimal folder layout:

direct-router/
  data/
    tasks.csv              # task_id, image_path, instruction
    planner_scores.csv     # task_id, planner_id, quality, cost
  routers/
    train_router.py
    infer_router.py
  planners/
    cheap_planner.py
    thinking_planner.py

tasks.csv can start with a few hundred scenes from a simulator or old robot logs. planner_scores.csv is the expensive part: run every planner on every task, then record quality and latency. In simulation, quality is success or progress. On hardware logs or offline datasets, you can use a rule-based evaluator if you have state labels, or an LLM judge if you only have the instruction and plan text.

The training loop is conceptually simple:

image_emb = siglip(image)
text_emb = bge_m3(instruction)
x = concat(image_emb, text_emb)

for planner in planners:
    y_quality[planner] = measured_quality(task, planner)
    y_cost[planner] = measured_cost(task, planner)

router.fit(x, y_quality, y_cost)

With scikit-learn, a RandomForestRegressor or MLPRegressor is enough for a first version. Predict a quality vector and a cost vector for every planner, then choose by utility:

utility = q_hat - lambda_cost * normalize(c_hat)
planner_id = utility.argmax()

Split train and validation by scene, not merely by rows. If the same scene appears in both train and validation with similar instructions, the router can overfit visual shortcuts and look better than it really is.

What to log during router training

You need enough logging to debug both planner behavior and router mistakes. At minimum, each planner call should store:

A common mistake is to train the router with a label such as "best planner" but not store cost. That turns routing into imitation of the expensive model rather than a quality-cost trade-off. DIRECT's core lesson is that you need both quality and cost. Set lambda_cost low if success is the dominant concern. Increase it when latency becomes a strict product requirement. In production, choose lambda_cost from a real SLA: can the robot wait 2 seconds, 5 seconds, or 20 seconds?

Inference in a robot stack

At runtime, DIRECT-style routing sits just before the high-level planner:

camera frames + current instruction
        |
        v
router.predict()
        |
        +-- cheap planner if the task is clear
        +-- thinking/large/memory planner if the task is hard
        |
        v
skill plan
        |
        v
low-level VLA / controller

For multi-step tasks, do not route once and keep the selected planner fixed for the entire episode. The paper's multi-step grocery bagging experiment shows why per-step routing matters: easy steps use no-thinking, while ambiguous steps escalate to thinking. This matches real robot operation because the scene changes after every pick or place. The global instruction may stay the same, but the next subgoal can have a different difficulty profile.

Deployment also needs guardrails. If the router selects the cheap planner but the returned plan is empty, calls a skill outside the allowlist, or has low confidence, fallback to a stronger planner. DIRECT focuses on efficient allocation; a product system still needs validation layers to block unsafe plans.

Main results

The paper reports more than 270,000 simulated routing decisions and 245 hardware trajectories across the three compute axes. On VLABench, DIRECT achieves the best routing efficiency in the compared thinking/non-thinking configurations, usually recovering near-expensive quality at lower latency. For model-size routing, cumulative routing over 2B, 4B, 8B, and 32B turns an erratic scaling curve into monotonic improvement; at the 32B point, the router adds 5.1 success points while reducing average latency by 32.4 seconds compared with selecting that model directly.

The clearest hardware result is multi-step grocery bagging: place fruits from heaviest to lightest into a white bin. Qwen3.5-VL 9B no-thinking reaches 47.62% success at 2.19 s latency. The thinking variant reaches 90.48% at 19.58 s. DIRECT reaches 95.24% at 6.85 s. In this setup, the router is both faster than the thinking planner and slightly better in success because it re-decides planner choice at each subgoal.

For memory routing on DROID, DIRECT approaches oracle performance more closely than the memory-free baseline and beats random or OOD routing, while processing fewer frames than the full memory planner. This supports the central claim: memory is compute, and compute should be allocated only when the task needs it.

When should you use DIRECT?

DIRECT is most useful when your lab already has multiple planners: a small fast model, a thinking model, a larger model, or a memory-augmented variant. If you only have one planner, routing cannot add capability. But if you have two or more options with different costs, DIRECT gives you a principled alternative to "always use the expensive model" or "always use the cheap model."

If you are new to the stack, start with our VLA Models overview. The OpenVLA deep dive explains how action policies are typically formed, while Embodied AI 2026 landscape places DIRECT in the broader foundation-model trend. DIRECT's novelty sits above the action policy: it allocates test-time compute for the planner.

Limitations

DIRECT is trained offline over a fixed planner pool. If you add a new model, change API latency, or move to different hardware, you should collect new quality-cost data and retrain the router. Cost is also deployment-specific: the same model may be cheap on a local GPU and slow through an API, or the reverse. Finally, the router only selects among existing planners. If none of the planners can command a required skill, routing cannot solve the task.

Still, the paper gives robotics teams a practical shift in thinking. Instead of asking only "what is the largest model we can afford?", ask "does this specific task need the large model?" For real robots, that question directly affects latency, cost, and how responsive the system feels.

Related Posts

• VLA Models in robotics
• OpenVLA deep dive
• Embodied AI 2026 landscape