Open Problems, Public Datasets, and Benchmarks in AV and Embodied AI

Sandbox brief for the Applied Intuition Data Intelligence team. The aim of this doc is to map the external scaffolding of the field — the public datasets, benchmarks, simulation/data standards, and regulatory frameworks that any AV or generalist-robotics team has to either consume, contribute to, or work around. Then, to enumerate the deep open problems that explain why the field still needs companies like Applied Intuition.

Last verified: 2026-05-08.

---

A. AV public datasets

Public AV datasets are how the academic community benchmarks itself and how every AV team bootstraps perception before touching their own (much larger) production logs. Two structural facts: (i) the largest open dataset (BDD100K, ~1,100 hrs) is a rounding error against Tesla's billions of fleet-miles or Waymo's tens of millions of rider-only miles; (ii) most are CC BY-NC research-only — the commercially-usable subset (parts of KITTI, ZOD, A2D2, BDD100K) is thin.

The takeaway: the public-AV stack has plenty of perception data, modest prediction data, very little closed-loop planning data, and almost no commercially-licensed data above 100 hours (ZOD is the exception). That gap is where vendor tooling sits.

---

B. Robotics public datasets

Robotics public datasets are 5–7 years behind AV in maturity, but the curve is steep. Open X-Embodiment (Oct 2023) was the watershed; AgiBot World Beta (Mar 2025) is plausibly the first robot dataset comparable in frame count to a large AV dataset.

Takeaway: every "X for robot learning" dataset has 1–2 OOM less data than the smallest commercial AV fleet collects in a week. Cross-embodiment is the lever, because no single embodiment has enough data alone.

---

C. Public benchmarks and leaderboards

Benchmarks turn datasets into competitions. The 2024–2025 trend in both AV and robotics has been a shift from open-loop (predict next waypoint, compare to log) to closed-loop (run the agent in sim and watch), because open-loop metrics correlate poorly with on-road performance.

C.1 AV leaderboards

• nuScenes detection / tracking / prediction (open-loop): still the most-submitted 3D perception leaderboard, NDS and AMOTA as headline metrics (nuScenes).
• Waymo Open Dataset Challenges — 6th annual edition in 2025, four tracks: Vision-Based End-to-End Driving, Scenario Generation, Interaction Prediction, Sim Agents. 2025 submissions closed May 22; leaderboards remain open (2025 blog). The new Rater Feedback Score for E2E compares predicted trajectories to human-rater preference labels rather than log replay — a meaningful methodological move.
• Argoverse Motion Forecasting (AV1: 324k scenarios; AV2: 250k) — minADE / minFDE / Miss Rate / brier-FDE on 6-mode multimodal forecasts.
• CVPR Workshop on Autonomous Driving (WAD) — 8th edition 2025, run by OpenDriveLab; hosts the Autonomous Grand Challenge with tracks for NAVSIM v2, Planning-Oriented AD, and E2E frontiers; ~$100k prize pool (CVPR 2025 OpenDriveLab).
• CARLA Leaderboard 2.0 — closed-loop sim eval. Driving Score = Route Completion × Infraction Penalty. Notoriously hard; most public submissions score sub-30.
• Bench2Drive (NeurIPS 2024 D&B) — built on CARLA L2.0. 220 short routes (5 × 44 scenarios × 23 weathers × 12 towns), 2M annotated training frames. First benchmark to disentangle multi-ability scoring (cut-in vs detour vs overtaking) so you can see where a stack fails (Bench2Drive).
• nuPlan planning challenge — closed-loop ML planning on real logs. Ran once in 2023; the devkit is no longer maintained by Motional (after Aptiv pulled back). Community forks (Horizon Robotics' is most active) and NAVSIM v2 (CVPR 2025) effectively supersede it (NAVSIM v2).

C.2 Robotics benchmarks

• Open X-Embodiment evaluations — no single OXE leaderboard; papers report on source-dataset splits + LIBERO + DROID held-out tasks. The fragmentation is itself a problem.
• LIBERO — the most-reported VLA benchmark in 2024–2025. Standard practice: success rate on Spatial / Object / Goal / Long. LIBERO-PRO (Oct 2025, arXiv 2510.03827) argues the original splits saturate and proposes harder variants.
• CALVIN — language-conditioned long-horizon manipulation in sim, 7-DOF Franka + desk; 24 hrs of teleop play, 20K language directives. Metric: instructions-completed-in-a-row (1 → 5).
• AgiBot World Challenge 2025, BEHAVIOR Challenge 2025, MimicBench / RoboBench / SimplerEnv / Robosuite — second-tier sim benchmarks; mostly for ablations.

The robotics-evaluation field is fragmented and contested — nothing yet with the cultural force of ImageNet-2012 or COCO. LIBERO is the closest, but it's sim-on-a-single-embodiment, and papers regularly disagree on what real-world success rates "should" look like.

---

D. Industry standards and regulatory frameworks

This is the layer that matters for shipping a real product, and where Applied Intuition has product-market fit — their tooling is the bridge between research-grade ML and ISO-26262-grade systems.

D.1 ASAM standards (simulation interop)

ASAM (Association for Standardization of Automation and Measuring Systems) maintains the open formats that let driving simulators and tool vendors interoperate. If you ship "scenario X to OEM Y," you express it in ASAM formats.

Together: describe a road, populate with actors, run in any compliant simulator, exchange labels. The foundation Applied Intuition's simulation product is interoperable on.

D.2 Functional safety and SOTIF

For ML-heavy AV systems, SOTIF is the more interesting standard — essentially "how to argue a probabilistic perception/planning stack is safe enough" — and it's the backbone of the AV safety case.

D.3 UN / UNECE WP.29 regulations

UN regs are the international type-approval framework (EU, UK, Japan, Korea, ~40 others; US uses FMVSS self-certification).

• UN R157 (ALKS) — adopted Jan 2021, in force Jan 2023. First international type-approval for an SAE L3 function. Original cap: 60 km/h; 01 Series Amendment raised it to 130 km/h (lane-change-capable systems only) (UNECE). 2024 supplement added EMC resilience.
• UN R155 (Cybersecurity / CSMS) — adopted 2020. Certified Cyber Security Management System mandatory for type approval. All new vehicles sold in UN-1958 contracting states from July 2024 must comply.
• UN R156 (Software Updates / SUMS) — R155 companion; governs the OTA lifecycle.
• WP.29 — the UN Working Party 29 body that drafts and amends the above through its GRVA and GRSG/GRSP working parties.

D.4 US regulatory framework

• NHTSA AV TEST Initiative (2020) — voluntary public reporting of testing locations and incidents.
• NHTSA AV STEP (NPRM Jan 15, 2025) — proposed ADS-equipped Vehicle Safety, Transparency, and Evaluation Program; voluntary, with independent assessment of safety cases (Federal Register).
• NHTSA AV Framework (April 24, 2025) — three principles: prioritize safety, remove barriers, enable commercial deployment (DOT).
• California DMV / CPUC — testing + deployment permits + ride-service licensing. As of late 2025, only Waymo holds a full deployment + driverless paid passenger permit; Zoox has driverless pilot permits. CA DMV released a revised regulatory proposal Dec 3, 2025 (CNBC).
• Texas SB 2807 (2025) — requires commercial AV operating authorization from Sept 1, 2025. Free, non-expiring.
• Arizona — ADOT self-certification; permissive. Waymo's largest ODD (~315 sq mi metro Phoenix); Tesla supervised-Robotaxi approval Nov 2025.

D.5 EU AI Act and other frameworks

The EU AI Act (adopted 2024, most provisions live Aug 2026) classes AVs as high-risk AI, but Article 25(4) routes them through sectoral type-approval law (UN regs + EU 2018/858 + EU 2019/2144) rather than the standard high-risk obligations. OEMs end up with two stacks: AI-Act-style documentation (data governance, bias monitoring, transparency logs) flowing through type approval (Squire Patton Boggs).

• SAE J3016 (J3016_202104, Apr 2021) — the canonical six-level (L0–L5) taxonomy. Joint SAE / ISO TC204/WG14 effort. The reference for "what level is this?" debates.
• PEGASUS (DE, completed mid-2019) — produced the scenario-based testing methodology now industry-standard (PEGASUS).
• ENABLE-S3 (EU, 2016–2019) — earlier validation-of-automated-systems work that fed into PEGASUS.
• Hi-Drive (EU Horizon, 2022–2026) — large-scale on-road testing across multiple EU countries; complements PEGASUS's sim focus with field data.
• SET Level (DE) — successor to PEGASUS for L4/L5 simulation pipelines.

---

E. Deep open problems

The standards above describe the rules of the game. The problems below are why the game isn't won.

E.1 The long-tail / safety-validation problem

The RAND "Driving to Safety" (2016) result: to demonstrate with 95% confidence that an AV's fatality rate is within 20% of the human baseline (1.09 fatalities per 100M VMT) requires 8.8 billion miles of testing; to show better than human, hundreds of billions (RAND RR-1478). Waymo's 170M cumulative rider-only miles are two OOM short. You cannot drive your way to safety. This is the structural case for simulation, scenario-based testing (PEGASUS, NAVSIM, Bench2Drive, Waabi World), and synthetic data — and the macro case for Applied Intuition. Corollary: edge-case discovery is its own discipline. CODA's <12.8% mAR for SOTA detectors is the headline that "pretrained on nuScenes" does not generalize.

E.2 The sim-to-real gap

Three sub-gaps: (1) sensor realism — does synthesized lidar/camera match physical noise statistics? (NVIDIA Cosmos, Waymo Sim, Waabi World, 3DGS reconstruction are converging on "yes for most, no for edge cases"); (2) behavior realism — do other agents drive like humans? This is the Sim Agents problem; 2025 winners hit ~0.785 realism against an unknown ceiling; (3) distribution realism — does the sim cover the long tail in the right proportion? Failure on any one collapses sim value.

E.3 Open-loop vs closed-loop evaluation

Open-loop ("predict 3 s, compare to log") is cheap and reproducible but doesn't compound errors — a planner drifting 1 cm/step looks fine open-loop and crashes closed-loop. Closed-loop is faithful but expensive, hard to standardize, and requires credible sim agents (see E.2). Bench2Drive and NAVSIM v2 are current best attempts at the middle. Waymo's RFS is a different angle: anchor to human preference, not log replay.

E.4 Data quantity vs quality (curation > collection)

Post-2024 consensus shifted from more to better. Tesla's shadow-mode triggers, Waymo's hard-mining, DROID's deliberate diversity curation, Open X's per-source quality filtering — all point the same way. Applied Intuition's Data Explorer / Axion reflect this: the rate-limiter isn't log volume, it's finding the 0.01% of frames that change model behavior.

E.5 Multi-agent realism

Single-vehicle planning in benign conditions is largely solved. Multi-agent interactive behavior — merging, unprotected lefts, occluded pedestrians, construction zones — is where stacks fail. The Sim Agents Challenge exists because without realistic NPCs, closed-loop sim is unfalsifiable. Arguably the hardest current AV sub-problem.

E.6 Generalization

Cross-city — a Phoenix stack doesn't work in Mumbai; Waymo's scaling law is HD-map + ODD + fleet-ops + weeks of fine-tuning per city. Cross-weather — fog, snow, heavy rain remain perception failure modes. Cross-embodiment (robotics) — Franka-trained policies don't transfer to xArm without retraining; Open X, RoboMIND's 4-embodiment design, and AgiBot World are explicit attacks on this.

E.7 End-to-end vs modular: the pendulum

• 2018–2022: modular wins (Waymo, Cruise, Aurora) — interpretable, debuggable, certifiable.
• 2022–2024: end-to-end wins (Tesla FSD v12 deleted ~300k lines of C++; Wayve / Waabi built E2E from day one).
• 2025–2026: convergence. Waymo's foundation model is "modular transformer-based" — separate transformers per stage, but each is a neural net. Tesla FSD v14 is fully E2E but uses auxiliary tasks for interpretability. Wayve's GAIA-2 (2025) pairs E2E with explicit safety filters. The dichotomy is dissolving toward "neural everywhere, modular interfaces for safety arguments." Applied Intuition's tooling survives the transition because the interfaces survive even as implementations go neural.

---

F. North-star goals across the field

Stand back and every research direction above aims at one of five long-term outcomes:

1. Robotaxi at scale. Driverless paid passenger service in arbitrary cities. Today: Waymo + Apollo Go + early Zoox / Tesla pilots. Open question: how fast can ODD expand?
2. Trustable end-to-end AV. A single neural net, demonstrably safe under SOTIF, certified under R157+, deployable at consumer scale. Today: Tesla FSD v14 (un-certified), Wayve, Waabi (trucking).
3. Generalist robot policy across embodiments. π0/π0.5, RT-2, OpenVLA — one policy controlling Franka, xArm, ALOHA, humanoids — given language.
4. True digital-twin sim. Near-zero domain gap on sensors, behavior, and distribution — enough to replace a meaningful fraction of real miles for both training and validation. Waabi World and the Waymo World Model (Feb 2026, built on DeepMind Genie 3) are the most credible attempts.
5. Data infrastructure that auto-converts drives/demos into training fuel. Closing the loop from "fleet log" → "labeled tensor" → "trained model" → "validated improvement" without human-in-the-loop bottlenecks.

(5) is a horizontal capability that all four other goals depend on. That is the Applied Intuition thesis.

---

G. Implications for Applied Intuition (and you)

Applied Intuition's portfolio (per the 2025 in-review post) covers:

• Simulation — ASAM-OpenSCENARIO/OpenDRIVE/OSI compliant, scenario authoring, sensor sim. (D.1)
• Data Explorer / Data Intelligence — fleet-log search, mining, curation, embedding retrieval. (E.4)
• Axion Tooling & Infrastructure — all-domain data engine + sim toolkit. (F.5)
• Acuity Ground / Air / Maritime — end-to-end autonomy stacks. (F.2 across non-passenger domains)
• ADAS (expansion 2025), Foundation models / voice / GenAI — adjacencies.

Highest-leverage gaps for the user's Data Intelligence team:

1. Closed-loop evaluation that's interpretable, auditable, and ASIL-mappable. Bench2Drive / NAVSIM v2 are research-grade; the productized OEM-safety-case version is wide open.
2. Multi-agent / sim-agents realism. The gating problem for sim-to-validate. Whoever ships realistic NPCs — data-driven, not heuristic — owns closed-loop testing. (Sim Agents Challenge is the research analog.)
3. Cross-embodiment data infrastructure for robotics. AI is mostly AV today, but the same primitives (search, curation, embedding retrieval, scenario mining, replay) are arguably more valuable in robotics where nobody has solved them. LeRobot is the OSS comparable; nobody has the enterprise version.
4. Long-tail / corner-case mining at fleet scale. Tesla's shadow-mode and Waymo's hard-mining are internal. Productized edge-case mining is a real wedge — and CODA's <12.8% mAR quantifies the headline claim.
5. Auto-labeling pipelines that survive SOTIF audit. AI's Mining massive autonomy datasets is already moving here — pretrained FMs auto-labeling fleet logs at human-level cost-per-frame. The bridge from "auto-labeled" to "ISO-21448-defensible" is the unsolved piece.

Thesis: the field has converged on the diagnosis ("data and evaluation are the bottleneck") much faster than the cure. Most public datasets and benchmarks codify the old regime (open-loop, log-replay, perception-heavy). The opportunity at AI Data Intelligence is to be in the room while the new regime — closed-loop, scenario-based, curation-first, multi-embodiment, SOTIF-defensible — is being defined and standardized.

---

Sources

• AV datasets: KITTI-360, Mapillary Vistas paper, nuScenes, nuPlan, Waymo Open, WOD-E2E, 2025 Waymo Open Challenges, Argoverse, Argoverse 2 paper, ZOD, ApolloScape, ONCE, A2D2, BDD100K, OpenLane-V2, CODA.
• Robotics datasets: Open X-Embodiment, DROID, BridgeData V2, RoboNet, BEHAVIOR-1K, LIBERO, AgiBot World, AgiBot World Beta on HF, RoboMIND, LeRobot, LeRobotDataset v3.0.
• Benchmarks: Bench2Drive, TrajTok 2025 Sim Agents, NAVSIM v2, LIBERO-PRO, CVPR 2025 OpenDriveLab, CALVIN.
• Standards: ASAM OpenSCENARIO 2.0 page, ASAM OpenSCENARIO DSL 2.1.0 spec, ASAM OpenDRIVE 1.9.0 spec, ASAM OpenLABEL, ISO 21448:2022 SOTIF.
• Regulation: UNECE R157 130 km/h amendment, Hogan Lovells on R155/R156, NHTSA AV STEP NPRM, DOT AV Framework April 2025, Squire Patton Boggs on EU AI Act for automotive, Volvo Autonomous on EU AI Act, SAE J3016_202104 (UNECE wiki copy), PEGASUS, California DMV AV permits.
• Research framing: RAND "Driving to Safety" RR-1478, Waymo World Model (Feb 2026), Waabi: The next frontier — generative AI in the physical world, Wayve Sequoia interview on AV 2.0.
• Applied Intuition: 2025 in review, AI for mining massive autonomy datasets, Series F at $15B valuation, Products page, OpenSCENARIO V2 explainer.