Project 19 — World-model-as-evaluator: can a WM judge a planner?

One-line goal: Take planner rollout clips with known closed-loop outcomes (from CARLA / Bench2Drive, or the bundled synthetic fallback), run a world-model-class judge over them (Cosmos Reason locally, or a frontier VLM via API, against a heuristic baseline), and measure how often the judge's verdict agrees with simulation ground truth. The deliverable is one number — "WM-eval agrees with sim-eval X% of the time" — plus the divergence taxonomy explaining the other (100−X)%.

This is the first of the two Round-3 additions to the portfolio (see /docs/08-connections-and-gaps.md §H''). It closes the triangle the other projects leave open: project 10 measures a world model, project 15 runs classical closed-loop eval — this project tests whether the world model can replace or pre-screen the classical eval.

Goal

By the end of this project you should be able to answer, with numbers:

Given a short driving clip, how accurately does a reasoning world model (or frontier VLM) detect that an infraction occurred — collision, near-miss, hard brake, lane invasion — compared to the simulator's ground-truth infraction log?
How well does the judge's scalar safety score rank rollouts compared to the closed-loop Driving-Score-style ranking? (Spearman correlation, not just accuracy.)
Where do judge and ground truth diverge, and is the divergence systematic (occlusions, slow-motion near-misses, "looks fine but violated a rule") or random?
Does the learned judge actually beat a dumb heuristic baseline (proximity + deceleration thresholds)? By how much?

The deliverables: agreement_pct (headline number), a per-infraction-type recall table, a judge-score vs ground-truth-score scatter with Spearman ρ, a divergence gallery with at least 5 annotated cases, and a written verdict on the three customer questions below.

Loops touched

EVAL — this is an eval-infrastructure project through and through: it evaluates the evaluator.
CURATE (secondary) — a judge that pre-screens millions of sim rollouts and routes only contested ones to expensive full re-simulation is a curation device. The agreement number you produce is exactly the routing threshold you'd tune.

Why this matters for AI Data Intelligence

Between late 2025 and mid 2026, "world models as evaluators" went from thesis to subfield: GAIA-3's entire framing is evaluation; the Waymo World Model is an eval substrate; Cosmos Reason is explicitly marketed as a rollout judge; and the spring 2026 papers (dWorldEval, CounterScene, the Interactive-World-Simulator line) attack policy evaluation inside learned world models. Every Applied Intuition customer will eventually ask some version of:

"Can we use a WM judge to triage our nightly regression runs so we only re-simulate the suspicious ones?"
"Can a WM judge score rollouts in environments we can't afford to model classically?"
"Should we trust a WM judge's verdict enough to gate a release on it?"

These have three different answers, and all three turn on the same measurement: agreement between the judge and a trusted closed-loop oracle, broken down by failure type. Nobody can responsibly answer "should we gate a release on it" (today: no) without knowing the per-type recall — a judge with 95% overall agreement that misses 40% of collisions is worse than useless, because it launders failures into green dashboards.

The deeper skill this project builds: treating an ML model as a measurement instrument and characterizing it the way you'd characterize a sensor — bias, variance, systematic failure modes, calibration. That is the core competency of an eval-infrastructure team.

Prerequisites

Python 3.10–3.12.
Comfort with structured-output prompting (project 05 builds this; its provider-SDK patterns are reused here).
Best path: completed project 15 (Bench2Drive) so you have real planner rollouts with infraction logs, or project 07 (CARLA parameter sweep). Fallback path: none — the bundled synthetic scenario renderer generates labeled clips so the full judging pipeline runs anywhere.
For the optional local-judge path: a Hugging Face token accepted into the Cosmos Reason gated repos.
For the API-judge path: an API key for at least one frontier VLM (Gemini / Claude / GPT).

Hardware

Path	Hardware	What you get
Heuristic judge + synthetic clips	Any laptop, CPU-only	The full pipeline + baseline numbers. Runs in minutes.
API VLM judge	Any laptop + API key	The headline experiment. ~$2–10 of API spend for 60–100 clips depending on provider and frame sampling.
Cosmos Reason local	24 GB+ GPU (Reason 2 8B at bf16 is tight on 24 GB; the 2B fits comfortably)	The "true WM judge" variant; compare against the API judge.
Cosmos Predict counterfactual probe (optional, Step 8)	H100/A100-80GB-class (see project 10)	The generative half: roll forward alternative actions and ask the judge to compare futures.

The notebook is structured so each path is independently runnable; GPU cells are flagged and skipped automatically.

Setup

cd projects/19-world-model-evaluator
bash setup.sh
source .venv/bin/activate

setup.sh creates a venv, installs requirements.txt, creates the output dirs, and — if it finds project 15 or project 07 outputs — symlinks them into ./data/. Set ANTHROPIC_API_KEY / GOOGLE_API_KEY / OPENAI_API_KEY in your environment for the API-judge path (the notebook reads whichever is present).

Steps

Assemble the evaluation corpus (# %% Step 1). Priority order: (a) project 15 route logs + infraction JSONs, (b) project 07 parameter-sweep captures, (c) the bundled synthetic scenario renderer — a 2D kinematic cut-in/lead-brake simulator that renders ~64 short BEV clips (MP4) with exact ground-truth outcome labels (clean / near-miss / hard-brake / collision) and a scalar ground-truth driving score. The synthetic path is not a toy: because you control the generative process, you know the labels are perfect, which is exactly what you want when characterizing a judge.
Define the verdict schema (# %% Step 2). A pydantic model: infraction: bool, infraction_type: enum, safety_score: 1–5, confidence: 0–1, rationale: str. Every judge must emit this schema — that's what makes them comparable.
Implement the heuristic baseline judge (# %% Step 3). Min-distance-to-actor + max-deceleration thresholds computed from the clip (synthetic path: from trajectory state; CARLA path: from the log). Every learned judge must beat this or it has no business existing. ~40 lines.
Implement the VLM API judge (# %% Step 4). Sample 6–10 frames per clip, send with a structured rubric prompt, parse the schema. Includes a cost tracker (pattern from project 05). Run over the full corpus, cache to outputs/verdicts/.
(Optional, GPU) Implement the Cosmos Reason judge (# %% Step 5). Same schema, same frames, local inference. Compare verdict distributions against the API judge before comparing either against ground truth — judge-vs-judge disagreement is itself informative.
Agreement analysis (# %% Step 6). Confusion matrix (judge infraction verdict vs ground truth), per-infraction-type precision/recall, agreement_pct, Spearman ρ between judge safety score and ground-truth driving score, and a calibration plot (judge confidence vs empirical accuracy). Save outputs/figures/agreement_matrix.png and outputs/figures/score_correlation.png.
Divergence gallery (# %% Step 7). Pull every disagreement, watch them, and tag each with a divergence category: judge missed it (e.g. glancing contact at clip edge), judge hallucinated it (hard-but-legal brake read as emergency), rule-vs-appearance (clip looks fine but violated a rule the judge can't see, like a speed limit), ground-truth artifact (the oracle label itself is debatable — it happens, write it down). At least 5 annotated cases in outputs/divergences/.
(Optional, big-GPU) Counterfactual probe (# %% Step 8). For 3 near-miss clips, use Cosmos Predict (project 10 infra) to generate two futures from the same mid-clip frame — the planner's action vs a braking action — and ask the judge which future is safer. This is the eval-side flip of generation and the closest thing to "WM-native evaluation" currently runnable on open models.
Write the verdict (# %% Step 9). Answer the three customer questions with your numbers. The template forces one sentence each on: triage use (viable at what threshold?), exotic-environment use, and release-gating use (expected answer: not yet — say why with your per-type recall table).

Done criterion

You are done when all of these exist:

outputs/tables/agreement.csv — per-judge: agreement_pct, per-type precision/recall, Spearman ρ. At least two judges (heuristic + one learned).
outputs/figures/agreement_matrix.png, outputs/figures/score_correlation.png, outputs/figures/calibration.png.
outputs/divergences/ with ≥ 5 annotated divergence cases, each tagged with a category.
The Step 9 verdict block filled in, including the explicit "would I gate a release on this judge?" answer justified by the per-type recall numbers.
The learned judge beats the heuristic baseline on agreement_pct — or, if it doesn't, a paragraph explaining why (that is a publishable-grade negative result; do not bury it).

Typical ranges to sanity-check against: frontier VLM judges land around 75–90% overall agreement on clean synthetic corpora, with collision recall high but near-miss vs hard-brake discrimination weak; Spearman ρ in the 0.5–0.8 band; heuristic baselines around 65–80% on synthetic data (they degrade much more on real CARLA logs, which is the point of running both corpora if you can).

Common pitfalls

Frame sampling decides the result. A judge that sees 6 uniformly-sampled frames can miss a 200 ms collision entirely. Sample denser around the kinematic peak (max |accel|) and say so in the write-up — and report sensitivity to the sampling rate.
Letting the judge see the answer. If your synthetic renderer draws a red flash on collision or your CARLA HUD shows infraction text, the judge is reading the label, not judging. Render clean.
Anchored rubrics. "Rate safety 1–5" without anchors produces provider-specific score distributions that destroy the correlation analysis. Anchor every point (1 = collision occurred … 5 = uneventful, comfortable). Same rubric verbatim for every judge.
Confusing agreement with correctness on a biased corpus. If 90% of your clips are clean, a judge that always says "clean" gets 90% agreement. Report per-type recall and balanced accuracy, and keep the corpus roughly balanced (the synthetic generator does this by construction).
Caching and determinism. API judges are not deterministic at temperature > 0. Set temperature 0, cache every verdict keyed by (clip-hash, judge, prompt-hash), and never re-judge silently — your numbers must be reproducible from outputs/verdicts/.
The oracle is only as good as the simulator. CARLA's infraction logger has known quirks (e.g., scraping a curb may not register). When judge and oracle disagree, watch the clip before assuming the judge is wrong — divergence category 4 exists for a reason.