When you read a new autonomous driving paper, your eyes will likely jump straight to the headline number: "Our method achieves 91.0 EPDMS on NAVSIM-v2 navtest." Impressive. But before you nod along, ask yourself: what kind of score is this, and what does it actually measure?
This post is a vaccine. The goal is not to introduce yet another benchmark — it's to teach you how to read benchmark numbers before trusting them.
The Core Problem: Three Score Types, Three Completely Different Meanings
In the autonomous driving world of 2026, at least three categories of evaluation scores coexist across leaderboards, papers, and technical blog posts. They often appear side by side in comparison tables but cannot be directly compared because they measure fundamentally different things:
| Score Type | Representatives | Question It Answers |
|---|---|---|
| Open-loop displacement | L2 error (m), ADE/FDE | Is the predicted trajectory close to the ground truth? |
| Pseudo closed-loop | NAVSIM PDMS / EPDMS | Does the planner violate traffic rules in simulation? |
| Long-tail / real-world | NAVSIM navhard EPDMS, WOD-E2E RFS | Can the planner handle rare, dangerous situations? |
If you only read the headline number, you will be misled. A model scoring 91 EPDMS on navtest can completely collapse on navhard — and that's exactly what happens in the example we'll analyze.
Type 1: Open-Loop L2 — Old Metric, Useful but Deceptive
L2 error (also known as Average Displacement Error — ADE, or Final Displacement Error — FDE) measures the average distance between the model's predicted trajectory and the trajectory the human driver actually took.
Advantages: easy to compute, easy to compare, no simulation required.
Fatal flaw: a model can fake excellence by copying historical trajectories. If the car is usually going straight, any model predicting "go straight" achieves a low L2 — even if it cannot handle an intersection at all.
Bench2Drive [NeurIPS 2024] put it plainly in their paper: "L2 error is not a meaningful indicator at all" for closed-loop evaluation. Their research showed that many models with low L2 scores repeatedly caused accidents when actually run in CARLA simulator.
When it's still useful: L2 is a valid sanity check — if a model has very high L2 (predictions off by several meters), that's a genuine red flag. But a low L2 guarantees nothing about safety.
Type 2: NAVSIM — Pseudo Closed-Loop (and the v1 vs v2 Distinction)
NAVSIM (NeurIPS 2024) was a major step forward: instead of comparing trajectories, it simulates whether the planner violates traffic rules over the 4 seconds following its prediction.
PDMS (v1) vs EPDMS (v2)
NAVSIM v1 used PDMS (Predictive Driver Model Score), aggregating sub-metrics:
- NC — No at-fault Collisions
- DAC — Drivable Area Compliance
- TTC — Time to Collision
- EP — Ego Progress (does the car actually move?)
- C — Comfort
NAVSIM v2 upgrades to EPDMS (Extended PDMS), adding:
- DDC — Driving Direction Compliance (correct side of road?)
- TLC — Traffic Light Compliance (obeying traffic lights?)
- LK — Lane Keeping (staying within lane boundaries?)
- HC — History Comfort (is the trajectory consistent with the vehicle's motion history?)
- EC — Extended Comfort (stable acceleration and jerk across frames?)
EPDMS also introduces smart false-positive filtering: if the human driver also violated the same rule, the penalty is disabled — preventing the model from being penalized for situations where no clean solution exists.
⚠️ Operational Trap #1: Branch v1 vs v2
This is the most important trap if you're setting up NAVSIM yourself:
- NAVSIM v2 lives on the
mainbranch of autonomousvision/navsim - NAVSIM v1 + the navtest leaderboard have been moved to the
v1.1branch
If you follow older tutorials (many blog posts and READMEs from 2024–2025 haven't been updated), you may install the main branch but expect v1 results — or vice versa. The leaderboard submission format also differs between versions.
navtest vs navhard — The Shocking Gap
NAVSIM v2 has two primary evaluation splits:
- navtest: Standard urban driving scenarios. Only Stage I (EPDMS₁). Relatively easy because the car is usually going straight or making simple turns.
- navhard: Long-horizon and safety-critical scenarios — unprotected turns, dense traffic, adverse conditions. Has two stages: Stage I (original scenario) + Stage II (5,462 observations synthesized from multiple endpoints of Stage I, weighted with a gaussian kernel).
Navhard simulates what safety engineers call long-tail events — situations that rarely occur but are extremely dangerous when handled incorrectly. This is why the score gap between navtest and navhard is typically enormous.
The Central Example: SUV and the 91.0 vs 36.9 Gap
The paper SUV: Future Scene Understanding as Video Generation for End-to-End Driving (arXiv:2608.03084v1, August 4, 2026, from Xi'an Jiaotong University, USTC, Yinwang Intelligent Technology, and Fudan University) is the perfect illustration of this problem.
SUV approaches autonomous driving by treating future scene prediction as a video generation problem: the model simultaneously generates 4 future streams — appearance, semantics, relative depth, and instance-level dynamics. A joint video-action attention mechanism allows the action expert to attend to latent representations of all four future streams for trajectory planning.
What makes it notable: SUV uses only a single front camera and no candidate-trajectory selection (the common approach of generating many candidate trajectories and picking the best one).
Leaderboard results:
NAVSIM-v2 navtest : 91.0 EPDMS ← Impressive
NAVSIM-v2 navhard : 36.9 EPDMS ← Reality check
WOD-E2E : 7.94 RFS
91.0 dropping to 36.9 — a 54-point gap. This is not a bad model; this is the fundamental nature of the long-tail problem.
The demo above shows an unprotected left turn — turning left without a protected signal, yielding to oncoming traffic. This type of scenario appears in navhard but is nearly absent in navtest. A model scoring 91.0 navtest is not guaranteed to handle it.
Core lesson: When reading a paper claiming NAVSIM SoTA, the first question is: navtest or navhard? If only navtest is reported, look for navhard. If navhard is absent, ask yourself why.
SimWAM: The PDMS-Latency Tradeoff
SimWAM (arXiv:2608.07468v3, repo H-EmbodVis/SimWAM) is the second example, focusing on a different dimension: the tradeoff between quality and inference cost.
SimWAM uses video generation purely as a training signal — after training, you discard the video branch entirely and only keep the action expert for inference. An isolated attention mask ensures action tokens cannot see future frames during prediction, so no video generation is needed at inference time.
Results:
NAVSIM navtest (PDMS) : 91.5 ← +0.5 above SUV
NAVSIM-v2 navtest : 90.2 EPDMS
NAVSIM-v2 navhard : 37.6 EPDMS ← +0.7 above SUV
Human agent baseline : 94.8 PDMS
The PDMS-vs-latency chart from SimWAM reveals something important: earlier world-model-based planners (DriveWAM, DriveLaW) had lower PDMS but significantly higher latency, because they had to generate video frames before extracting actions. SimWAM breaks that tradeoff by decoupling the training and inference phases.
Component ablation from the paper:
- Action-only baseline: 86.6 PDMS
-
- Video co-training: 90.3 PDMS (+3.7)
-
- Reinforcement learning: 91.5 PDMS (+1.2)
This is a useful example of how to read ablation studies: video generation doesn't directly improve inference quality — it improves the learned representations during training.
Bench2Drive: When Closed-Loop Is Actually Closed-Loop
NAVSIM is still pseudo closed-loop — the car doesn't actually interact with the environment, and other agents don't react to the ego vehicle's actions. Bench2Drive (NeurIPS 2024 Datasets and Benchmarks Track, repo Thinklab-SJTU/Bench2Drive) is the next step: real simulation in CARLA 0.9.15.
Bench2Drive evaluates several distinct driving abilities:
- Driving Score: primary performance metric
- Success Rate: task completion percentage
- Multi-Ability Results: overtaking, merging, emergency braking, yielding, traffic sign recognition
Version 0.0.4 standardizes training data to 44 scenarios × 25 routes (1,100 routes total), ensuring balanced scenario distribution. The test set contains 220 official routes.
Why is Bench2Drive harder than NAVSIM? Because other vehicles in CARLA actually react to the ego vehicle. If you brake suddenly, the car behind changes behavior. If you move into the left lane, oncoming traffic has to swerve. NAVSIM does not do this — other agents are "frozen" in replay.
⚠️ Operational Trap #2: CARLA Leaderboard and AD Challenge
- The current CARLA Leaderboard is version 2.1 (from March 2025). Results from Leaderboard 2.0 or 1.0 are not directly comparable.
- The CARLA Autonomous Driving Challenge (AD Challenge) was not held during 2025. Many papers cite "CARLA Challenge" but are actually running on private setups or old versions.
CLEAR: RL in CARLA — Reference, Not Tutorial
CLEAR: Closed-Loop Reinforcement Learning at Scale for End-to-End Autonomous Driving (arXiv:2607.02841v1, Qualcomm AI Research) is one of the most interesting 2026 papers on closed-loop RL.
CLEAR trains a residual waypoint policy built on top of a pretrained Vision-Language-Action model, using RL to fine-tune it in CARLA longest6 v2 and Bench2Drive environments. They address sample efficiency with a heterogeneous computing pipeline enabling parallel CARLA environments.
Results: state-of-the-art on both CARLA longest6 v2 and Bench2Drive.
But one important fact must be stated clearly: CLEAR has no public code and no images I could verify. This is a good reference paper for understanding methodology, but if you want to reproduce it or build on top of it — you'll need to implement from scratch. I mention it here because it illustrates the direction of closed-loop RL, not as a setup guide.
WOD-E2E: Waymo's Measurement Axis
WOD-E2E (Waymo Open Dataset — End-to-End track) uses the RFS (Relative Frequency Score) metric. Unlike NAVSIM, RFS evaluates whether the model can reproduce the behavioral distribution of human drivers on real-world data from Waymo's fleet.
SUV achieves 7.94 RFS on WOD-E2E — a number without context unless you know what the human baseline is. Critically, WOD-E2E includes many long-tail situations from Waymo's actual data, not CARLA simulation.
⚠️ Operational Trap #3: Waymo Has No Challenge in 2026
Waymo Open Dataset did not organize an official challenge in 2026, but the leaderboard is still active — you can still submit and receive rankings. Don't confuse "no competition" with "leaderboard is no longer relevant."
Summary: A Checklist for Reading Any Paper
Here's a quick checklist for reading any autonomous driving paper in 2026:
When you see: "91.x EPDMS/PDMS on NAVSIM"
→ Ask: navtest or navhard? If only navtest → find navhard
When you see: "SoTA on CARLA"
→ Ask: CARLA Leaderboard 2.1? CARLA 0.9.15? longest6 v2?
Or did they run a private CARLA with no standard?
When you see: "NAVSIM" but code has issues
→ Ask: Using main branch (v2) or v1.1 branch (v1)?
Is the submission format the same?
When you see: Low L2 error, impressive
→ Ask: Did they run any closed-loop evaluation?
Low L2 ≠ good driving
When you see: "RFS on WOD-E2E"
→ Ask: No 2026 Waymo challenge but leaderboard is live
RFS = relative to human distribution, not absolute
Three Score Types — Reading Papers Like a Senior Engineer
To summarize, here's how a senior engineer reads a results table:
If only open-loop L2/ADE/FDE → The paper is reporting trajectory imitation ability, not safe driving ability. Ask about closed-loop.
If there's NAVSIM navtest EPDMS → Good, but find navhard. The gap between the two numbers reveals much about the model's robustness.
If there's navtest + navhard + WOD-E2E → This is a paper serious about deployment. The model is stress-tested on multiple axes: common scenarios, dangerous long-tail situations, and real behavioral distributions.
If there's Bench2Drive Driving Score → The model has been tested in true closed-loop with reactive agents. This is the closest to reality possible without a real vehicle.
Next in the Series
Post 4 dives into NVIDIA Alpamayo — a reasoning-based model from NVIDIA with a distinctive architectural approach to autonomous driving. After reading this post, you now have the "immunity" to read Alpamayo's results table without being swayed by a single headline number.
