On this page
Glossary — Physical AI Terms and What They Actually Mean
One-page (well, one-file) reference for every recurring term in this repo. Each entry gives the short definition first, then the context that makes it useful in conversation. Terms are grouped by theme, not alphabetized — the groups mirror the four-loop frame from 00-overview.md. Use Ctrl-F.
Last verified: 2026-06-12.
A. The big frame
Physical AI — The body of techniques, products, and infrastructure for getting learned policies to act safely and competently in the physical world. The two visible verticals are autonomous vehicles and humanoid/manipulation robotics; the shared substrate is the data infrastructure (collect → curate → label → train → eval → deploy → re-collect).
Data engine / data flywheel — Karpathy's framing: deploy a model, mine its failures, re-label and retrain with those failures included, redeploy. The insight that mattered: the infrastructure to mine and curate the right 1% of incoming data is harder and more valuable than the infrastructure to train the model. "The data engine is the product" is now industry consensus; the fight is over whose data engine becomes the system of record. See 04 §C.
The four loops — COLLECT (fleet logs, sim, world-model generation — weeks–months), CURATE (triage, mining, dedup — hours–days), LABEL & TRAIN (auto-label, verify, distill — days–weeks), EVAL (open-loop, closed-loop sim, coverage, safety case — continuous). Every project in this repo is tagged against these.
Long tail — The rare events (a stroller in rain, a mattress on the freeway, an emergency vehicle parked at a strange angle) that dominate safety risk despite being a vanishing fraction of miles. Perception is not solved on the long tail — SOTA detectors score <12.8% mAR on the CODA corner-case benchmark.
Shadow mode — Running a candidate model passively alongside the deployed system and logging where they disagree. Tesla's flywheel staple: disagreements are uploaded as training candidates. The general form is disagreement mining.
Trigger-conditioned upload — On-device classifiers ("emergency vehicle present", "policy disagrees with driver") that decide which clips a consumer fleet uploads. The bandwidth-efficient way to harvest the long tail from millions of cars.
B. AV stack and operations
ODD (Operational Design Domain) — The bounded set of conditions (geography, weather, time, road types, speeds) within which an autonomy system is designed to operate. L4 means "driverless within the ODD." ODD taxonomies are also the slicing schema for eval ("find all night-time unprotected lefts in rain").
SAE Levels (L0–L5) — The J3016 taxonomy. L2 = hands-on/eyes-on assistance; L2+ /L2++ = marketing terms for better L2; L3 = conditional automation (driver is fallback, e.g. UN R157 ALKS); L4 = driverless within an ODD (Waymo); L5 = driverless everywhere (nobody).
Robotaxi — Paid, driverless (no safety driver) passenger service. The operating players as of mid-2026: Waymo, Baidu Apollo Go, Pony.ai, WeRide, Zoox, and Tesla's monitored pilots.
ADAS (Advanced Driver-Assistance Systems) — The L1–L2++ consumer feature tier (lane keeping, adaptive cruise, supervised city driving). The volume market: tens of millions of vehicles/year, dominated by Mobileye, Nvidia DRIVE, Qualcomm, Huawei ADS, BYD God's Eye, Horizon Robotics.
Modular stack vs end-to-end (E2E) — Modular: separate perception → prediction → planning → control modules with human-designed interfaces (classic Waymo). End-to-end: one learned network from pixels to trajectory (Tesla FSD v12+, Wayve). The 2026 synthesis: "neural everywhere, modular interfaces for safety arguments."
HD map — Centimeter-grade prior map (lane geometry, connectivity, signals) used for localization and as a planning prior. Map-light/map-free is the E2E camp's counter-position. REM is Mobileye's crowdsourced HD-map flywheel: customer cars upload ~10 KB/km of compressed semantic landmarks.
Disengagement — A safety driver (or remote operator) taking control from the autonomy stack. Disengagement-cause labels are a regulatory reporting requirement (CA DMV) and a core eval signal.
BEV (bird's-eye view) — The dominant intermediate representation for AV perception since ~2022: project multi-camera (and lidar) features into a top-down ego-centric grid, then detect/predict in that space. BEVFormer and BEVFusion are the canonical architectures (project 04).
Occupancy network — Perception output representing 3D space as occupied/free voxels (plus flow), rather than discrete object boxes. Popularized by Tesla; ground truth comes from accumulated lidar or offline reconstruction, which makes the auto-label pipeline the hard part.
Sensor calibration: intrinsics / extrinsics — Intrinsics: a camera's internal geometry (focal length, distortion). Extrinsics: the 6-DoF pose of each sensor relative to the ego frame. Multi-sensor labels are only as good as the calibration; drift is a constant production headache.
Ego pose / ego frame — The vehicle's own position and orientation over time, and the coordinate frame attached to it. Everything in an AV log is ultimately registered to ego pose.
C. Labeling and curation
Auto-labeling — Machine-generated labels, typically from a much larger offline "teacher" model that sees the full log including future frames (non-causal context). Production reality since ~2023: most labels are auto-generated and human-verified, not human-drawn. The canonical references are Tesla's offline 4D pipeline and Waymo's Auto4D / offboard-3D papers.
Offboard / offline perception — The heavyweight multi-pass models used for auto-labeling (as opposed to the real-time onboard model). The onboard model is distilled from offboard outputs — "model-as-teacher."
Labels from the future — Behavior/trajectory labels extracted by replaying the log: "the car did change lanes 3 s later, so lane_change_intent=True." Free by construction, but only available in retrospect.
Open-vocabulary detection / segmentation — Detecting or segmenting classes specified by free-text prompt at inference time, rather than a fixed taxonomy. Grounding DINO (text → boxes), SAM/SAM 2 (prompt → masks), and SAM 3 (text → masks directly) are the workhorses.
Grounded-SAM — The default first-pass auto-label pipeline: text prompt → Grounding DINO boxes → SAM 2 masks → temporal propagation across video.
VLM-as-labeler — Using frontier vision-language models (Gemini, GPT, Claude) for semantic/behavioral labels ("is the pedestrian about to cross?") that previously required humans. The 2025–26 labeling frontier; project 05 builds the decision matrix for when it beats traditional pipelines.
Embedding mining — Embed every clip (CLIP/SigLIP for text-aligned queries, DINOv2 for visual nearest-neighbor), index with ANN (FAISS/Qdrant), then query by text or by example — especially K-NN around known failures. The most useful 5-line tool in curation.
CLIP-text query — "Find clips that look like 'a child running between parked cars at dusk'." Productized in FiftyOne Brain, Scale Nucleus, Encord Active.
Active learning — Choosing which unlabeled samples to label next for maximum model improvement. Core methods: uncertainty sampling (label lowest-confidence), BALD (mutual information via MC-dropout), coreset / k-center (maximize diversity), BADGE (gradient-embedding clustering — uncertainty × diversity). The inner loop of every data engine (project 16).
Weak supervision / labeling functions — Snorkel's paradigm: write programmatic heuristics that each label noisily; a generative model resolves their conflicts into probabilistic labels. The alternative to "more humans."
Pseudo-labels / self-training / noisy student — Train, predict on unlabeled data, keep high-confidence predictions as labels, retrain. The cheapest label multiplier.
Cross-modal label propagation — Label in the easy modality (2D camera mask via SAM), project through calibration into the hard one (lidar segmentation). Dominant technique for "free" 3D labels.
IAA (inter-annotator agreement) — How much independent labelers agree: Cohen's κ (two raters), Fleiss' κ (many), Krippendorff's α (general), mean IoU (geometric). Production teams gate annotators on per-class IAA floors (κ ≥ 0.8 is typical).
Mistakenness / uniqueness — FiftyOne Brain's per-sample scores: likelihood a ground-truth label is wrong (from model disagreement), and how distinct a sample is from the rest of the dataset. Vocabulary for "find the label errors" and "dedup the boring highway miles."
Data provenance / lineage — The audit trail of where every training sample came from and what transformed it. SOTIF and the EU AI Act make this non-optional; world-model-generated data makes it urgent (project 06).
PII redaction — Blurring faces and license plates before logs enter training. Every commercial AV pipeline does this (EgoBlur is the open default).
RLHF / RLAIF / preference labels — Fine-tuning from human (or AI) rankings of output pairs. The AV/robotics version: humans rate 10-second driving clips or manipulation rollouts for comfort/quality. The labeling category expected to grow as E2E stacks adopt the LLM post-training playbook.
D. Simulation and synthetic data
Scenario-based testing — Evaluating against a parameterized library of situations ("cut-in at TTC 1.2 s in rain") rather than raw mileage. The methodology PEGASUS standardized and Applied Intuition's Simian productizes. Scenario coverage — the open question "have we tested enough of the space?" — is the regulatory choke point.
OpenSCENARIO / OpenDRIVE / OpenLABEL / OSI — The ASAM standards: scenario description (XML 1.x; DSL 2.x for parametric scenarios), road-network description, annotation format, and the simulator↔system-under-test interface, respectively. The lingua franca with OEM customers — standards literacy is graded in vendor evaluations.
SIL / HIL / VIL — Software-, hardware-, vehicle-in-the-loop: running tests against the software stack alone, against the real ECU hardware, or with a real vehicle on a rig/track.
Log replay / re-simulation — Re-running recorded drives against a new software version to catch regressions; the cheap, grounded end of the sim spectrum. Replay-and-perturb edits actor trajectories in a real log to make counterfactuals.
Open-loop vs closed-loop evaluation — Open-loop: predict the next seconds, compare to the log; cheap but errors don't compound (a planner drifting 1 cm/step looks fine and crashes in reality). Closed-loop: the policy's actions actually steer the simulation; faithful but requires credible reactive agents. The field's decisive shift in 2024–26 is from open- to closed-loop (Bench2Drive, NAVSIM v2).
Sim agents / NPCs — The simulated other road users. If they don't behave like humans (reactive, negotiating, occasionally aggressive), closed-loop sim is unfalsifiable. The Waymo Sim Agents Challenge (WOSAC) is the canonical benchmark; realism has plateaued around ~0.78–0.79.
Sim-to-real gap (sim2real) — The performance drop between simulation and reality, decomposed into sensor realism, behavior realism, and distribution realism. Domain randomization is the classic counter: randomize physics/lighting/textures in sim until reality is just another sample.
Digital twin — A reconstructed, re-simulatable copy of a real place or scene. Real-to-sim is the pipeline that builds it from logs.
NeRF / 3D Gaussian Splatting (3DGS) — Neural scene representations built from camera (± lidar) frames. 3DGS represents the scene as explicit 3D Gaussians that render in real time, are editable, and can be recomposed — which made "drive → reconstruct → re-simulate with new actors" feasible (Wayve PRISM-1, EmerNeRF, Street Gaussians; project 08). 4DGS adds the time dimension for dynamic scenes.
Hybrid sim — The pattern that actually won: classical simulation for ground-truth labels, physics, and determinism + a generative model (Cosmos Transfer-style) for photoreal appearance and visual diversity. Projects 07 + 09 build it end-to-end.
Synthetic data multiplier — Generating many training variants from one real seed (MimicGen-style replay for robot demos: 10×–1000×; Cosmos Transfer re-styling for AV frames). The economics that make small real datasets viable.
E. World models
World model — Overloaded; five distinct families: (1) latent dynamics models (Dreamer, MuZero — tiny, no pixels, for RL-in-imagination); (2) generative video models (Sora, Veo — pixels, no action interface); (3) driving world models (GAIA, Cosmos-Drive — multi-camera, action/layout-conditioned); (4) robotics world models (1X WM, GR00T-Dreams — action-conditioned egocentric video); (5) JEPA-style (V-JEPA — predicts in latent space, non-generative). Knowing which family someone means is half the conversation.
WFM (world foundation model) — Nvidia's term for large pretrained world models meant to be post-trained per domain (the Cosmos family: Predict = generate futures, Transfer = controlnet-style re-styling of structured inputs, Reason = VLM judge of physical plausibility).
Action conditioning — The difference between "make a video of a car turning left" and "given THIS scene, show what happens if I steer +5°." What turns a video generator into a world model. Action-conditioning drift — the gap between commanded and realized trajectory in generated video (~1–2 m lateral over 5 s at the 2025–26 frontier) — is the metric that gates closed-loop use (project 10).
Object permanence / identity drift — Signature world-model failure modes: an occluded car returns as a different car; 4 cyclists become 3 then 5.
JEPA (joint-embedding predictive architecture) — LeCun's counter-bet: predict future representations, not pixels — don't waste capacity on texture. V-JEPA 2 demonstrated zero-shot robot planning from ~62 hours of robot data. Its limitation is also its point: you can't render training data out of it.
World-model-as-evaluator — Using a WM to re-drive logged scenarios with controlled variations (GAIA-3's framing), or a reasoning model to judge rollouts (Cosmos Reason). The eval-side flip of "WM as data engine."
F. Robotics and VLAs
VLA (vision-language-action model) — A policy that maps pixels + a language instruction to robot actions. The lineage: RT-1 → RT-2 → RT-X/OpenVLA → π0/π0.5/π0.7, Helix, Gemini Robotics, GR00T. The dominant recipe for generalist manipulation.
Action tokenization / action head — How a VLA emits actions: discrete tokens (RT-2), diffusion policy heads (Octo, GR00T), flow matching (π0 — continuous action chunks at 50 Hz), or frequency-space FAST tokens (π0-FAST). Action chunking = predicting a short sequence of actions per inference step.
System 1 / System 2 architecture — The two-tier humanoid pattern: a slow VLM reasons (7–9 Hz), a fast visuomotor transformer controls (200 Hz), communicating via a latent vector (Figure Helix; GR00T N1). Helix 02 added a kHz "System 0" balance layer.
Embodiment — A specific robot body (kinematics, sensors, action space). Cross-embodiment training pools data across many robots; positive transfer is established for similar arms, weak for hands and legs. Embodiment normalization — mapping robots into a canonical action/observation schema — is the unsolved infrastructure problem.
Teleoperation (teleop) — Humans piloting robots (VR, leader-follower arms like ALOHA, mocap suits) to record demonstrations. The dominant data source for VLAs; ~$30–50/hr loaded, 50–150 demos/hr — the economics that make synthetic multipliers existential.
Behavioral cloning (BC) — Supervised learning on (observation, expert-action) pairs. The base recipe for both VLAs and E2E driving; the hidden labeling problem is filtering for genuinely expert demonstrations.
Open X-Embodiment (OXE) — The pooled cross-embodiment dataset (~1M+ trajectories, 22 embodiments, 60 datasets) that is robotics' "ImageNet moment" claim. RLDS is its episode schema; LeRobotDataset (Hugging Face) is the de-facto open format for new data (v3.0 added streaming).
Egocentric video — First-person human video (Ego4D, Project Aria) harvested as free supervision for manipulation. The catch: no action labels, no force feedback.
Sim benchmarks: LIBERO / CALVIN / RoboCasa / BEHAVIOR-1K — The VLA evaluation suites. LIBERO is the most-reported; LIBERO-PRO showed SOTA models collapsing from >90% to ~0% under modest perturbations — the field's reminder that benchmark pass-rates ≠ generalization.
RaaS (Robots-as-a-Service) — Paying per robot-hour/task instead of buying hardware. Agility×GXO (2024) was the first multi-year humanoid RaaS contract.
G. Safety, standards, regulation
ISO 26262 — Functional safety for automotive E/E systems: hazard analysis, ASIL (Automotive Safety Integrity Level A–D) classification, protection against system failures.
SOTIF (ISO 21448) — Safety Of The Intended Functionality: risk from functional insufficiencies without any fault — e.g., an ML perception model failing on glare. The standard that governs "how do you argue a probabilistic stack is safe enough," and the reason auto-labels need provenance, calibration, and uncertainty metadata.
ISO/SAE 21434 / UN R155 / UN R156 — Automotive cybersecurity engineering, the cybersecurity management system regulation, and the software-update (OTA) regulation.
UN R157 (ALKS) — The first international type-approval regulation for an L3 function (Automated Lane Keeping); speed cap raised 60 → 130 km/h. The template for how L3/L4 certification will look in UNECE markets.
V&V (verification & validation) — The discipline of demonstrating a system meets requirements (verification) and is fit for purpose (validation). Applied Intuition's Validation Toolset / Test Suites productize it. Safety case — the structured argument, backed by evidence artifacts, that a system is acceptably safe for deployment.
The RAND result — "Driving to Safety" (2016): demonstrating AV fatality rates statistically requires ~8.8 billion test miles. You cannot drive your way to safety — the structural argument for simulation and scenario-based validation.
EU AI Act — Classes AVs as high-risk AI but routes them through sectoral type-approval; adds data-governance/traceability documentation obligations (Aug 2026 milestone, partial deferral under debate).
FMVSS / NHTSA SGO — US federal vehicle safety standards (self-certification regime) and NHTSA's Standing General Order requiring crash reporting for ADAS/ADS vehicles.
H. Data formats and plumbing
MCAP — Foxglove's container format for multimodal robot/AV logs; the emerging default over ROS bags. ROS 2 / rosbag2 — the robot middleware and its native recording format. Foxglove Studio — the standard log-visualization tool (project 02).
Protobuf / schema registry — How message types are defined and evolved inside log formats; schema drift across a fleet is a real curation problem.
ANN index (FAISS, ScaNN, Qdrant, Milvus) — Approximate nearest-neighbor search over embedding vectors; the backbone of petabyte-scale curation queries.
Jupytext percent format — .py notebooks with # %% cell markers, used for every project notebook in this repo: native in VSCode, diffable in git, convertible to .ipynb.
NDS / mAP / mAR / minADE / minFDE — Headline metrics: nuScenes Detection Score (composite 3D detection), mean Average Precision (detection), mean Average Recall (the CODA long-tail metric), minimum Average/Final Displacement Error (motion forecasting, over k modes).
Driving Score (CARLA) — Route Completion × Infraction Penalty; the closed-loop headline metric in CARLA Leaderboard 2.0 / Bench2Drive.
I. Company/product shorthands used in this repo
Simian / Spectral / Data Explorer / Validation Toolset / Neural Sim / Axion — Applied Intuition's scenario sim, sensor sim, log curation, V&V backbone, real-to-sim generative sim, and all-domain data-engine toolkit, respectively.
Cosmos / Isaac Sim / Isaac Lab / GR00T / NeMo Curator / Omniverse — Nvidia's WFM family, robotics simulator, RL training framework, humanoid foundation-model line, GPU data-curation library, and digital-twin/rendering substrate.
FiftyOne (Voxel51) — The open-source dataset curation/visualization library; the lingua franca of vision data engineering and the closest OSS analog to Data Explorer.
GAIA (Wayve) — The driving world-model line (GAIA-1 → 2 → 3); GAIA-3's framing is evaluation, not just generation. PRISM-1 — Wayve's camera-only 4D reconstruction (real-to-sim).
Waymax — Waymo's JAX-based sim-agents research simulator built on the Open Motion dataset (project 14).
LeRobot (Hugging Face) — The open robot-learning library + dataset hub; the closest thing to "GitHub for robot data."