Self Driving Car Sensors: How Cars See the Road

Here’s a question I love asking people: how many eyes does a self-driving car have? Most people guess one or two — a camera, maybe. The real answer is somewhere between a dozen and forty, depending on the car, and they’re not all the same kind of eye. A self-driving car sees the world through cameras, lasers, radar, and sound, all at once, and the story of how those senses combine is one of the most interesting engineering tales in modern technology.

This guide walks through every major type of self driving car sensor — what it measures, what it’s good at, where it fails — and then the real magic: sensor fusion, the process of merging all those streams into one coherent picture of the world. There’s also the industry’s great religious war: cameras versus lidar, Tesla versus everyone else. For the full picture of what the car does with all this sensing, see my overview of how self-driving cars work.

Why One Sensor Is Never Enough

Every sensor is a tradeoff. Cameras see detail but fail in darkness and glare. Lidar measures distance precisely but struggles in heavy rain and costs a fortune. Radar works in any weather but has poor resolution — it can tell you something is there but not whether it’s a car or a metal sign.

The principle is redundancy through diversity. You don’t want two sensors that fail in the same conditions; you want sensors whose weaknesses don’t overlap. When the camera is blinded by sunset glare, the radar doesn’t care. When rain scatters the lidar, the cameras still see lane markings. The car’s perception system is designed so that no single sensor failure — and no single bad condition — leaves it blind.

Cameras: The Car’s Eyes

Cameras are the workhorses. A typical autonomous vehicle carries anywhere from 6 to 12 of them, positioned to give 360-degree coverage: forward-facing long-range cameras for the road ahead, wide-angle cameras for intersections, side cameras for lane changes, rear cameras for reversing.

What cameras are brilliant at: reading the world the way humans do. Traffic lights, road signs, lane markings, brake lights, turn signals, the facial orientation of a pedestrian deciding whether to cross — all of this is visual information, and cameras capture it in rich detail. A camera can tell a stop sign from a yield sign, a red light from a green one, a child from a trash bag. No other sensor does this.

What cameras are terrible at: measuring distance directly (a camera image is flat — depth has to be inferred), seeing in darkness, handling glare and sudden lighting changes, and seeing through rain, snow, or fog. A camera blinded by low sun is as useless as you squinting into it.

Lidar: Painting the World With Lasers

Lidar (Light Detection and Ranging) is the sensor that looks like the future. It fires millions of laser pulses per second and measures how long each takes to bounce back, building a precise 3D point cloud of the environment — a shimmering dot-matrix model of every car, curb, pedestrian, and lamppost within range, accurate to centimeters.

What lidar is brilliant at: distance and shape. It doesn’t guess how far away something is; it measures it, directly, hundreds of times per second. It works in complete darkness (it brings its own light). The point cloud it produces is the closest thing the car has to “seeing” solid geometry.

What lidar struggles with: cost (though prices have fallen dramatically — from $75,000 units a decade ago to hundreds of dollars today), weather (heavy rain and snow scatter the laser pulses), and reading visual information (it can’t read a traffic light’s color or a sign’s text — the point cloud has no color, or limited color).

Radar: Seeing Through the Weather

Radar is the unglamorous hero. It uses radio waves to detect objects and measure their speed via the Doppler effect, and it does this reliably in rain, fog, snow, and darkness — exactly the conditions where cameras and lidar struggle.

What radar is brilliant at: velocity and all-weather reliability. Radar tells you not just that there’s a car ahead, but how fast it’s moving relative to you, instantly. That’s why adaptive cruise control — one of the oldest driver-assistance features — was built on radar. It’s cheap, proven, and nearly indestructible.

What radar is bad at: resolution. Traditional automotive radar has a blurry view of the world — it can struggle to distinguish a stopped car from an overhead sign, or a pedestrian from roadside clutter. This “low resolution” problem caused real incidents in early driver-assistance systems. Newer imaging radar (sometimes called 4D radar) dramatically improves resolution, and it’s one of the quieter technology upgrades happening in the industry right now.

Every serious autonomous vehicle carries radar. It’s the sensor that keeps working when everything else is having a bad day.

Ultrasonic Sensors: The Parking Whiskers

Ultrasonic sensors are the small discs in your bumpers — the ones that beep faster as you back toward a wall. They use sound waves to measure very short distances, typically under 10 meters.

They’re not glamorous, but they’re essential for the near field: parking, low-speed maneuvering, detecting the curb you’re about to scrape. Autonomous systems use them the same way a cat uses whiskers — for the space immediately around the body, where the big sensors’ view is blocked by the car itself.

GPS, IMU, and HD Maps: Knowing Where You Are

Sensing the world is only half the problem; the car also needs to know where it is in the world. Standard GPS gets you within a few meters — good enough to know which street you’re on, useless for knowing which lane.

Autonomous vehicles layer several systems: high-precision GPS (sometimes with real-time corrections), an IMU (inertial measurement unit — accelerometers and gyroscopes that track motion when GPS drops, like in tunnels), and HD maps — centimeter-accurate pre-mapped models of the roads the car drives.

HD maps are the controversial one. Robotaxi companies like Waymo map their operating areas exhaustively, down to the curb heights, and the car localizes itself against that map. It’s enormously effective within the mapped area and useless outside it — which is exactly why robotaxis are geofenced. Tesla’s approach famously skips HD maps, betting that the car should drive like a human: reading the road as it comes. Both approaches work, partially, and the debate continues.

Sensor Fusion: The Real Magic

Here’s where it gets genuinely interesting. Having four kinds of sensors is useless if they disagree — and they disagree constantly. The camera says the lane curves left; the lidar point cloud is noisy from rain; the radar reports a phantom object. Sensor fusion is the art and science of combining these streams into one coherent, trustworthy model of the world.

Modern fusion works at multiple levels. Low-level fusion combines raw data (lining up the camera image with the lidar point cloud, pixel by pixel). Mid-level fusion combines detected objects (“the camera sees a pedestrian at these coordinates, the radar confirms something moving at walking speed there”). High-level fusion reconciles the conclusions (“three sensors agree there’s a stopped truck ahead; the fourth is an outlier — trust the majority”).

The fusion system also tracks objects over time — not just “there’s a car” but “there’s a car, it’s been in that lane for four seconds, it’s decelerating, and based on its blinker it’s probably merging.” This temporal tracking is where how self-driving cars use AI comes in: neural networks predict what each object will do next, and the car plans around those predictions.

Fusion is also where redundancy pays off. The system is designed so that sensors vote, and the car trusts the consensus — with careful rules for breaking ties. A well-designed fusion system degrades gracefully: lose the lidar, and the car drives more cautiously on cameras and radar rather than going blind.

The Great Debate: Cameras vs Lidar

Now the religious war. Tesla’s position: cameras alone (plus radar, at times, then not, then again) are sufficient, because humans drive with vision alone, and lidar is an expensive crutch. Almost everyone else’s position: that’s reckless — direct depth measurement is a safety-critical redundancy, and the cost keeps falling.

Meanwhile, the lidar camp sometimes understates how good modern vision has become. Learned depth estimation from cameras is startlingly accurate now, and sensor fusion with radar covers many of the gaps.

What the Sensors Still Can’t Handle

For all this technology, the sensor suite has known blind spots — and I mean that literally:

Adversarial weather. Not rain — heavy, sensor-defeating downpours, whiteout snow, dense fog. Every sensor degrades; fusion degrades with them. Most autonomous systems simply refuse to operate in these conditions, which is the correct engineering answer and an unsolved problem.

Unusual objects. The perception system is trained on common things. A mattress in the road, a person in a costume, an overturned truck with an unusual profile — these are the long tail of weirdness that testing struggles to cover.

The “unknown unknowns.” The fundamental limit: sensors report the world as it is, but driving requires understanding the world as it will be — the pedestrian’s intention, the other driver’s distraction. Sensors give you the present; prediction is inference, and inference can be wrong.

None of this means the sensors don’t work — they work remarkably well, in the conditions they’re designed for. It means the operational envelope is defined as much by sensing limits as by anything else. When a robotaxi won’t drive in a storm, it’s not being cautious for show. It’s telling you the truth about its eyes.

For the complete story of what happens after the sensors see — perception, prediction, planning, control — read my full guide on how self-driving cars work.

Frequently Asked Questions