How Do Self-Driving Cars Work? Sensors, AI, and Reality Check

Self-driving cars have been “five years away” for about fifteen years now. I’ve followed this technology closely — read the papers, watched the disengagement reports, ridden in the robotaxis that actually operate — and the gap between the demo videos and the daily reality is the most interesting part.

So how do self-driving cars work, really? Let’s go through the actual technology: what the car sees with, how it thinks, and where things genuinely stand in 2026.

For a deeper dive, see our guide to using AI at work.

Contents

• The Five Levels (What “Self-Driving” Actually Means)
• What the Car Sees With: The Sensor Suite
• How the Car Thinks: Perception to Action
• The HD Map Question
• Where Things Stand in 2026
• Why It’s So Hard
• Should You Trust One?

The Five Levels (What “Self-Driving” Actually Means)

First, terminology — because marketing abuses it relentlessly. The industry standard (SAE levels):

• Level 0–1: Warnings and single assists (lane departure warning, basic cruise control)
• Level 2: The car steers AND accelerates/brakes, but you must watch constantly and stay ready. This is Tesla Autopilot, GM Super Cruise, Ford BlueCruise. This is not self-driving, whatever the ads imply.
• Level 3: The car drives itself in specific conditions, and you can look away — but must take over when asked. Almost nobody sells this yet.
• Level 4: Fully driverless within a defined area and conditions. Waymo robotaxis operate here in several cities.
• Level 5: Drive anywhere, any conditions, no steering wheel needed. Doesn’t exist. May never exist in the pure form.

When someone says “self-driving car,” ask which level. The answer is usually Level 2 wearing a Level 5 costume.

What the Car Sees With: The Sensor Suite

A self-driving car perceives the world through overlapping sensors — redundancy is deliberate, because each sensor type has blind spots:

Cameras. The primary eyes. Modern systems use 8–12 cameras giving 360° vision. Cameras are cheap, high-resolution, and great at reading signs, lights, and lane markings. Weakness: they struggle in darkness, glare, and heavy rain — exactly when you’d most want help.

LiDAR. Shoots millions of laser pulses per second and builds a precise 3D map from the reflections. It’s the most accurate depth sensor available — it “sees” the exact shape and distance of everything. Weakness: expensive (though prices have crashed from $75,000 to under $1,000 per unit), and degraded by heavy rain and fog.

Radar. Uses radio waves to measure distance and speed of objects. Lower resolution than LiDAR, but works through rain and fog, and directly measures velocity — incredibly useful for knowing that the car ahead is slowing down.

Ultrasonic sensors. Short-range parking sensors. Boring, reliable, been in cars for decades.

The industry split worth knowing: Waymo and most others use the full suite (cameras + LiDAR + radar). Tesla bets on cameras alone, arguing humans drive with just eyes so cars should too. I’ve examined both approaches’ safety records, and the honest assessment is that the jury is still out — but redundancy strikes me as the obviously safer engineering choice.

How the Car Thinks: Perception to Action

The software pipeline, simplified:

1. Perception. Neural networks process sensor data to answer: what’s around me? This means detecting vehicles, pedestrians, cyclists, lane lines, traffic lights, signs — and tracking how each moves over time. Modern systems do this with deep learning models trained on millions of miles of driving footage.

2. Prediction. For each detected road user, predict what they’ll do next. That pedestrian glancing at their phone near the curb — are they about to step out? This is genuinely hard AI; humans do it with social intuition that machines approximate with statistics.

3. Planning. Given the predicted world, plan a safe, comfortable path: which lane, what speed, when to merge, how to handle the double-parked delivery truck. This blends learned behavior with hard-coded safety rules.

4. Control. Convert the plan into steering, throttle, and brake commands, executed hundreds of times per second.

The whole loop runs continuously. A Waymo vehicle processes this pipeline roughly 10 times per second — slower than the control loop, faster than human reaction time for most situations.

The HD Map Question

Here’s something the demo videos don’t mention: most robotaxis don’t just drive by sight. They use HD maps — centimeter-accurate 3D maps of their operating area, built in advance by mapping vehicles.

The car localizes itself within this map (knowing its position to within centimeters), then uses live sensors mainly to detect changes — construction, parked cars, pedestrians. This is why Waymo works brilliantly in mapped parts of Phoenix and can’t just drive to an unmapped town.

It’s a legitimate approach, but it means “self-driving” currently means “self-driving where we’ve pre-mapped every inch.” Scaling that to everywhere is one of the industry’s biggest unsolved challenges.

Where Things Stand in 2026

The honest state of play:

• Robotaxis work, in limited areas. Waymo operates driverless rides in multiple US cities with a safety record that, by the published data, compares favorably to human drivers in those areas. This is real, not a demo.
• Highway assists are good. Level 2 systems from several manufacturers handle highway driving competently — but require constant supervision, and misuse kills people.
• Full autonomy everywhere isn’t close. Unmapped areas, bad weather, construction zones, and chaotic urban environments remain unsolved or barely solved.
• The business is brutal. Most self-driving startups have died or been acquired. The survivors are burning billions. The technology works better than the business model does.

I’ve ridden in a Waymo. It’s genuinely impressive — smooth, cautious, competent. It’s also geofenced to sunny mapped streets. Both facts are true simultaneously, and anyone selling you only one of them is selling something.

Why It’s So Hard

Driving looks easy because humans are good at it, but it requires general intelligence applied continuously: reading a construction worker’s hand gestures, knowing that a ball rolling into the street means a child may follow, understanding that the hesitant driver ahead is lost.

These are “long tail” problems — rare situations, each different, collectively common. Machine learning excels at common patterns and struggles with rare ones. Driving is 99% common patterns and 1% weird situations that can kill you. That last 1% is the entire difficulty of the problem.

This is why I tell people: the question was never “can AI drive?” It’s “can AI handle everything driving throws at it, including the things that happen once in a million miles?” We’re at “mostly, in mapped sunny areas.” The rest is still research.

The Ethics Questions Nobody’s Fully Solved

The technology works well enough to deploy, which raises questions the industry is still wrestling with — and I think you should know them as someone who might ride in or buy these vehicles.

The trolley problem is real now. If a crash is unavoidable, how should the car choose? Protect occupants at all costs? Minimize total harm? These aren’t philosophy-class hypotheticals anymore — they’re parameters someone has to set in code. Manufacturers don’t publish their answers, which tells you something about how comfortable they are with the question.

Liability is legally murky. When a human driver crashes, liability is clear. When a Level 4 robotaxi crashes with no driver, who’s responsible — the passenger, the fleet operator, the software company, the sensor manufacturer? Courts are deciding this case by case, and the legal framework is years behind the technology.

Data and surveillance. These vehicles record everything — high-resolution video of public streets, continuously. That’s necessary for the technology, and it’s also the most comprehensive mobile surveillance network ever built. Who accesses the footage, how long it’s kept, and what else it’s used for are questions with unsatisfying answers right now.

Jobs. Millions of people drive for a living. Autonomous trucks and robotaxis won’t replace them overnight, but the transition — when it comes — will be economically brutal for affected workers. The technology conversation rarely includes them; it should.

I don’t raise these to scare you away from the technology. I raise them because “how do self-driving cars work” includes how they fit into society, not just how the sensors work. The engineering is impressive. The societal integration is just beginning.

Should You Trust One?

My practical guidance:

• Robotaxis in operating areas: the published safety data is genuinely good. I’d ride in one — and have.
• Level 2 assists: useful, but treat them as cruise control that steers, not autopilot. The name “Autopilot” has arguably killed people through overtrust. Hands on wheel, eyes on road, always.
• Buying a car “for” self-driving: don’t. Buy the car for what it does today. Future software updates are promises, not features.

The technology is real, improving, and already saving lives in limited deployments. It’s just not the revolution the 2016 keynotes promised — it’s something slower, more careful, and ultimately more trustworthy for being so.

Frequently Asked Questions