Project 20 — Preference data and reward modeling for driving (RLHF for E2E)

One-line goal: Generate paired driving trajectories of the same scenario under different driving styles, collect preference labels through three channels (a hidden synthetic oracle, your own A/B judgments, and optionally a VLM judge), train a Bradley–Terry reward model on the pairs, and measure (a) held-out pairwise accuracy, (b) how label noise degrades it, and (c) inter-annotator agreement between you, the oracle, and the VLM.

This is the second Round-3 addition to the portfolio (see /docs/08-connections-and-gaps.md §H''). It is the project that most directly converts LLM-era post-training skills (RLHF, preference data, reward models) into physical-AI credibility.

Goal

By the end you should be able to answer, with numbers:

1. How many pairwise preference labels does it take for a small reward model to rank driving trajectories at >85% held-out pairwise accuracy against the labeling source?
2. How fast does that degrade as labels get noisier (oracle with 0% / 10% / 25% label flips — the realistic human-rater regime per docs/01 §D.2's "humans disagree 10–25% on hard cases")?
3. When you, the oracle, and a VLM label the same pairs, what is the inter-annotator agreement (Cohen's κ), and what kinds of pairs produce disagreement?
4. What did the reward model actually learn? (Permutation feature importance: is it jerk, TTC margin, lane discipline — and is that what the labeling source was actually rewarding?)

Deliverables: outputs/figures/accuracy_vs_labels.png (the headline learning curve, with noise bands), outputs/tables/iaa.csv (the κ matrix), a ranked gallery of best/worst trajectories per the learned reward, and a write-up paragraph on what this implies for "comfort raters" as a data category.

Loops touched

• LABEL & TRAIN — preference labels are labels; the reward model is the trained artifact. This is the AV analogue of the RLHF data pipeline described in docs/04 §D (reward/preference labels).
• EVAL (secondary) — a learned reward model is also a continuous evaluator: it scores any new rollout. The ranked-gallery step makes that concrete.

Why this matters for AI Data Intelligence

docs/09 §C.7 predicts every Tier-1 AV stack gains an RLHF post-training stage by 2027, with "comfort raters / ride-experience experts" becoming a recognized data category parallel to LLM RLHF annotators. The evidence keeps gathering: PE-RLHF in the literature, Tesla's preference-tuning loop, Wayve's preference data, and the labeling market's hard pivot toward expert preference work (Surge/Mercor economics, docs/04).

The infrastructure questions a Data Intelligence team will own are exactly the ones this project makes you feel:

1. Label-source routing. Synthetic oracles are free but encode your utility function; humans are expensive and noisy; VLM judges are cheap and biased differently. When do you use which? You will have a κ matrix that says how much they actually disagree.
2. Label-budget economics. The accuracy-vs-labels curve is the same curve as project 16's active-learning plot, but for preference data — and it directly prices "how many rater-hours does a comfort model cost?"
3. Reward hacking and feature audit. A reward model that secretly keys on trajectory length instead of comfort will happily reward planners that stop early. The permutation-importance step is the audit that catches this — the physical-AI version of LLM reward-model overoptimization.

Note what this project is not: it does not RL-fine-tune a planner against the learned reward (that is a different, much heavier project). Fitting and auditing the reward model is the data-infrastructure half — and the half the role owns.

Prerequisites

• Python 3.10–3.12.
• Comfort with PyTorch training loops (project 16 level).
• Nothing else. The trajectory generator is bundled; CARLA is optional (the generator deliberately mirrors project 07's cut-in scenario so a CARLA-sourced corpus can be swapped in later).
• Optional: an API key (Anthropic / Google / OpenAI) for the VLM-judge labeling channel — same provider plumbing as projects 05 and 19.

Hardware

Laptop, CPU-only. The reward model is a small MLP over trajectory features; full runs (3 noise levels × 6 budget points × 3 seeds) take ~10–20 minutes on a modern laptop. No GPU path needed.

Setup

cd projects/20-driving-preference-reward
bash setup.sh
source .venv/bin/activate

Steps

1. Generate the trajectory corpus (# %% Step 1). The bundled generator simulates the project-07-shaped cut-in scenario under a style-parameterized ego controller (headway, max accel, brake comfort, lane-keeping discipline). ~400 trajectories across ~100 scenario seeds × 4 style presets (timid / smooth / assertive / aggressive). Each trajectory records the full kinematic state sequence.
2. Extract features (# %% Step 2). Per trajectory: RMS jerk, max |accel|, max |decel|, min TTC, min gap, RMS lane-center offset, mean speed vs desired, progress. These are the candidate explanations the reward model can use — keep the list small enough to audit.
3. Define the hidden oracle (# %% Step 3). A weighted comfort+safety utility over the features. It is "hidden" in the sense that the reward model never sees it — only pairwise comparisons derived from it. You will compare what the model learned against these weights at the end.
4. Build preference pairs (# %% Step 4). Sample pairs within the same scenario seed (same traffic, different ego style — this is the disciplined comparison; cross-scenario pairs confound style with situation difficulty). Generate oracle labels at three noise levels: 0%, 10%, 25% random flips.
5. (Recommended) Label ~40 pairs yourself (# %% Step 5). The notebook renders side-by-side BEV animations and collects your A/B choice. Do not skip this if you can spare 20 minutes — the visceral experience of "these two look the same to me" on hard pairs is the point, and it is your κ data.
6. (Optional) VLM-judge labels (# %% Step 6). Send the same ~40 pairs to a frontier VLM with an anchored comfort rubric (provider plumbing reused from project 19). Third κ channel.
7. Train the reward model (# %% Step 7). A 2-layer MLP over features, trained with the Bradley–Terry pairwise loss -log σ(r(A) − r(B)) on the preferred-A pairs. Train across the label-budget grid (25 → 800 pairs) × noise levels × 3 seeds.
8. Evaluate (# %% Step 8). Held-out pairwise accuracy curves (the headline figure); Spearman ρ between learned reward and oracle utility on held-out trajectories; permutation feature importance vs the oracle's true weights.
9. IAA analysis (# %% Step 9). Cohen's κ between every pair of label channels on the commonly-labeled pairs; a table of the pairs with maximal disagreement, with your notes on why (close calls, style taste, rubric ambiguity).
10. Ranked gallery + write-up (# %% Step 10). Score a fresh batch of trajectories with the learned reward; render the top-3 and bottom-3; fill in the reflection block — including the "comfort raters as a data category" paragraph and the reward-audit verdict.

Done criterion

You are done when all of these exist:

1. outputs/figures/accuracy_vs_labels.png — pairwise accuracy vs number of labeled pairs, one curve per noise level, with seed variance bands. Typical shape: >85% by 150–300 clean pairs; the 25%-noise curve plateauing visibly lower.
2. outputs/tables/iaa.csv — the κ matrix across {oracle, you, VLM} on ≥ 30 commonly-labeled pairs (or {oracle, you} if you skipped the VLM channel). Expect κ(you, oracle) in the 0.4–0.7 band — write down whether your disagreements were noise or a genuinely different utility.
3. outputs/figures/feature_importance.png — permutation importance next to the oracle's true weights, plus one sentence on any mismatch (that mismatch is reward mis-specification, named).
4. outputs/gallery/ — top-3 and bottom-3 trajectories per the learned reward, rendered.
5. The Step 10 reflection block filled in.

Common pitfalls

1. Cross-scenario pairs. Comparing a trajectory in light traffic against one in a brutal cut-in measures scenario difficulty, not driving quality. Pair within scenario seed, always. (Production systems use matched-context pairs for exactly this reason.)
2. Leaky features. If progress dominates the learned reward, your model rewards "got further" — which on fixed-duration clips just means "drove faster," and your "comfort model" is a speed model. Check importance before believing accuracy.
3. Degenerate pairs. Pairs where both trajectories are near-identical (two smooth runs) teach nothing and depress your κ. The notebook filters pairs below a minimum feature distance; report how many you dropped.
4. Labeling yourself while knowing the styles. The side-by-side renderer shuffles A/B and hides the style preset. Don't peek at the params before choosing — anchoring is real and it will inflate your κ with the oracle.
5. BT loss on logits, not probabilities. The Bradley–Terry loss takes raw reward differences through a sigmoid; if you sigmoid the rewards first, training still "works" and the calibration is silently wrong.
6. Reading too much into κ with n=40. The confidence interval on κ at 40 pairs is wide (~±0.2). The notebook prints it; quote it when you quote the κ.

Further reading

• Christiano et al., Deep RL from Human Preferences (2017) — the origin of the BT-loss-on-trajectory-pairs recipe you implement here: https://arxiv.org/abs/1706.03741
• Anthropic, Constitutional AI — the RLAIF lineage the VLM-judge channel belongs to: https://arxiv.org/abs/2212.08073
• PE-RLHF (2025) — physics-enhanced RLHF for driving, the closest published AV instance: https://www.sciencedirect.com/science/article/abs/pii/S0968090X25002669
• docs/04 §D — reward/preference labels and §E (IAA metrics, multi-pass review) — the production framing.
• docs/09 §C.7 — the "comfort raters become a data category" prediction this project tests in miniature.
• Project 16 (active learning) — the label-budget twin of this project; the natural extension is preference-pair active learning (sample the pairs the reward model is least sure about).