You know how a sensor makes points. Now: how a car turns a swarm of points into knowing where it is and what to do next — SLAM, sensor fusion, and the representations that carry meaning. （定位、感知融合、点云表示）
Every self-driving system is a pipeline （流水线）. Raw sensor data enters one end; a steering/throttle decision comes out the other. Tap a stage to see its job.
Classically these are separate modules wired in sequence — modular （模块化）. The 2024–25 trend is collapsing them into one trainable network — end-to-end （端到端）. More on that at the end.
A point cloud （点云） is just an unordered list of (x, y, z) coordinates — maybe with intensity （反射强度）. That "unordered" part is the whole challenge.
An image is a tidy grid — pixel (0,0), (0,1)… a network always knows where each value sits. A point cloud has no grid, no order, no fixed count. Shuffle the points and it's the same scene, so the math must give the same answer regardless of order (permutation invariance （排列不变性）). Points are also sparse （稀疏） — far objects get only a handful of hits. Toggle the views below to feel the difference.
Each dot is one laser return. Drag to orbit — notice it's a thin shell on surfaces, not a solid.
Networks can't eat raw points directly at scale, so we reshape them. Tap each to expand.
Feed raw points straight into a network (PointNet-style) that's built to ignore order. Keeps full geometric fidelity — nothing is rounded off.
Snap points into a 3D grid of little cubes called voxels （体素 = 3D 像素）. Now ordinary 3D convolutions work. This is what most LiDAR detectors use under the hood.
Flatten height and look straight down — a Bird's-Eye View （鸟瞰图）. Cars never stack vertically, so a top-down map loses little and makes planning easy. The dominant representation today.
SLAM = Simultaneous Localization and Mapping （同步定位与建图）. The chicken-and-egg problem at the heart of robotics: to map the world you must know where you are, but to know where you are you need a map.
SLAM solves both at once. As the car moves, it matches each new scan against the last (scan matching （扫描匹配）), estimates how far it moved, and stitches the scans into a growing map. Press play and watch a map assemble from nothing.
Tick drift to see the core failure mode: tiny per-scan errors accumulate and the map bends. Real SLAM fights this with loop closure （回环检测） — recognizing a previously-seen place and snapping the whole map back into alignment.
Odometry （里程计） — "how far did I move since last frame?" Drift-prone, no memory. Localization （定位） — "where am I on a map I already have?" SLAM — do both, with no map to start. Robotaxis often skip live SLAM by pre-building a centimeter-accurate HD map （高精地图）, then just localizing against it.
No sensor is complete. Fusion （传感器融合） combines them so each covers the others' blind spots. The question is where in the pipeline you merge them.
| Capability | LiDAR 激光 | Camera 摄像头 | Radar 毫米波 |
|---|---|---|---|
| 3D distance | excellent | inferred | coarse |
| Color / text / signs | none | excellent | none |
| Velocity | computed | inferred | direct |
| Rain / fog / snow | degrades | degrades | robust |
| Night / darkness | unaffected | poor | unaffected |
| Cost | ~$200+ | cheap | cheap |
← swipe · green = strength, red = weakness →
This table is the argument for fusion: every column has a red cell, and no two sensors are red in the same row. That complementarity （互补性） is why most L3+ systems carry all three — and why the camera-only bet is contrarian.
The frontier you care about. Perception is moving from "list the objects" toward "predict the whole scene as a volume" — and then toward simulating it forward in time.
Instead of fitting bounding boxes around known classes, the car divides space into a 3D grid and asks each cell: occupied or free? and by what? This handles the weird stuff a box-detector misses — debris, an overturned truck, a mattress on the highway — because it never has to name the object to avoid it. It's 3D semantic scene completion （语义场景补全） in real time.
Modern stacks lift every camera and LiDAR feed into one shared bird's-eye grid, then let a transformer's attention （注意力） fuse them and reason across the whole scene at once. LiDAR gets BEV almost for free by dropping the height axis; cameras need a learned 2D→3D lift, which is the hard, active research part.
An occupancy world model （占用世界模型） doesn't just perceive the current volume — it rolls it forward, predicting how the occupied scene evolves over the next few seconds. That's a learned simulator of the physical world, the same idea powering neural simulation for AV training. It's exactly the bridge from perception into the simulation / world-models work you're aiming at: the point cloud stops being a snapshot and becomes the input to a predictive model of reality.
Modular: separate, debuggable blocks (detect → track → predict → plan), each individually testable but with hand-built interfaces that lose information between stages. End-to-end: one network from sensors to steering, trained jointly — higher ceiling and less information loss, but harder to interpret and validate. The field is converging on end-to-end with occupancy/BEV as the shared internal representation. Validating these black boxes is precisely where high-fidelity simulation becomes indispensable.
| Term | 中文 | One-line meaning |
|---|---|---|
| SLAM | 同步定位与建图 | Build a map and locate yourself in it at the same time |
| Odometry | 里程计 | Estimate motion frame-to-frame; drifts over time |
| Loop closure | 回环检测 | Recognize a revisited place to correct accumulated drift |
| Point cloud | 点云 | Unordered set of 3D points from a sensor |
| Voxel | 体素 | A 3D pixel; cube cell in a spatial grid |
| BEV | 鸟瞰图 | Top-down view used as a shared scene representation |
| Occupancy network | 占用网络 | Predict free/occupied for every cell in 3D space |
| Sensor fusion | 传感器融合 | Combine multiple sensors into one estimate |
| HD map | 高精地图 | Centimeter-accurate prior map of roads & lanes |
| Permutation invariance | 排列不变性 | Answer must not depend on point ordering |
| End-to-end | 端到端 | One network from raw sensors to driving action |
| World model | 世界模型 | Learned simulator that predicts how a scene evolves |
← swipe to see meanings →