How Do Voice Assistants Process Speech? The Full Pipeline

Every voice assistant interaction — “set a timer,” “what’s the weather,” “play my workout playlist” — travels through the same seven-stage pipeline. Microphone to meaning to action to answer, in about a second. The underlying pipeline is shared across all of them.

Think of this as the master schematic. Once you see the stages, you can pinpoint any voice assistant failure (“it heard me wrong” versus “it understood me wrong” are different stages failing). For the fundamentals of the recognition stage, see my guide on how voice recognition works.

Table of Contents

• The Seven Stages at a Glance
• Stage 1: Capture (From Sound Waves to Signal)
• Stage 2: Wake-Word Detection (The Bouncer)
• Stage 3: Speech Recognition (Sound to Text)
• Stage 4: Language Understanding (Text to Meaning)
• Stage 5: Dialogue Management (Deciding What Happens Next)
• Stage 6: Action (Doing the Thing)
• Stage 7: Response Generation (Meaning Back to Sound)
• Where the Pipeline Runs: Device, Edge, Cloud
• How Errors Cascade Through the Pipeline
• The Pipeline Is Collapsing (in a Good Way)
• Frequently Asked Questions

The Seven Stages at a Glance

Here’s the full journey of “Hey Siri, what’s the weather in Chicago?”: capture, wake-word detection, speech recognition, language understanding, dialogue management, action, and response. Each stage is a distinct AI problem with its own models and failure modes.

Stage 1: Capture (From Sound Waves to Signal)

It starts with physics. Microphones convert air pressure variations into electrical signals, sampled thousands of times per second (typically 16,000 samples/second for voice). That raw signal is a mess — it contains your voice plus everything else: room echo, background noise, the device’s own speaker output, wind, handling noise.

Signal processing cleans it up before any AI sees it:

Beamforming combines multiple microphones to focus on your direction. It’s the audio equivalent of a camera zooming — except it’s done with math, steering a “listening beam” toward your mouth.

Echo cancellation subtracts the device’s own output. When your smart speaker is playing music and you talk over it, the system knows exactly what it’s playing and removes that signal from what the microphones hear. Without this, voice assistants couldn’t hear you while making noise themselves.

Noise suppression uses neural networks trained to separate speech from non-speech — suppressing a barking dog or passing truck while preserving your voice.

Automatic gain control normalizes volume — whisper or shout, the downstream stages get consistent levels.

The quality of this stage sets the ceiling for everything after.

Stage 2: Wake-Word Detection (The Bouncer)

The wake-word detector is the pipeline’s bouncer: small, fast, always on, and extremely specialized. It runs continuously on low-power hardware, watching the audio stream for one specific pattern — “Hey Siri,” “Alexa,” “OK Google.”

The model is tiny by design — it has to run 24/7 without draining the battery. It doesn’t understand language; it recognizes an acoustic signature. Think of it as a highly trained ear that knows one sound and ignores everything else.

The engineering tradeoff is false accepts versus false rejects. Tune it sensitive and it wakes up during TV shows (annoying). Tune it strict and it misses you when you mumble (infuriating). Every vendor picks a point on this curve, and every user has opinions about whether they picked right.

The bouncer doesn’t just open the door; it hands off a recording of you approaching it — the buffered audio from before the trigger, so the start of your request isn’t lost.

Stage 3: Speech Recognition (Sound to Text)

Automatic speech recognition (ASR) converts the captured audio into text. This is the stage most people mean by “voice recognition,” and it’s where the heaviest AI lifting happens.

Modern ASR uses end-to-end neural networks: acoustic features go in, text comes out. Under the hood, the model has learned the mapping from sound patterns to words across millions of hours of training speech. It handles accents, speaking rates, and disfluencies through sheer statistical exposure — it’s heard enough variation to generalize.

The output isn’t just a single transcript — it’s a ranked list of hypotheses with confidence scores. Downstream stages use these confidences to decide whether to act or ask for clarification.

ASR can run on-device (Apple’s approach — private, fast, offline-capable) or in the cloud (Amazon’s approach — bigger models, more compute, needs connectivity). The architectural choice shapes the product’s privacy, latency, and capability profile more than any other single decision.

Stage 4: Language Understanding (Text to Meaning)

Text is not meaning. “Set a timer for ten minutes” is a string of characters; the assistant needs to extract the actionable structure: intent=set_timer, duration=10 minutes. This is natural language understanding (NLU), and it’s a completely different problem from recognition.

NLU has two sub-tasks:

Intent classification — what does the user want? The system maintains a catalog of supported intents (set timer, play music, get weather, send message, control device…) and classifies the utterance into the best match. Modern classifiers handle paraphrase well: “wake me up in ten,” “set an alarm for ten minutes from now,” and “timer, ten minutes” all land on set_timer.

Entity extraction (slot filling) — what are the details? From “play jazz on Spotify,” extract genre=jazz, service=Spotify. From “text Mom I’ll be late,” extract recipient=Mom, message=”I’ll be late.” The entities plug into the intent’s schema like arguments to a function.

This stage is where “the assistant is dumb” complaints usually originate. The NLU only understands intents in its catalog — ask for something outside it and the system has nowhere to put your meaning. It’s not failing to think; it’s failing to categorize.

Stage 5: Dialogue Management (Deciding What Happens Next)

Between understanding and acting sits the dialogue manager — the pipeline’s strategist. Given the understood intent and entities, it decides what to do next: execute immediately, ask a clarifying question, or handle an error.

Slot completion: “Set a timer” with no duration → “For how long?” The dialogue manager tracks what’s missing and asks for it naturally.

Confirmation: “Delete all my photos” → “Are you sure?” High-stakes actions get a confirmation step. The dialogue manager encodes the product’s risk judgment.

Context tracking: “What’s the weather?” → “In Chicago, 72 and sunny.” → “What about tomorrow?” The manager resolves “tomorrow” against the established context (Chicago, weather). Multi-turn conversation lives here.

Error recovery: Low recognition confidence → “Sorry, I didn’t catch that.” The manager decides when to admit failure versus when to guess.

This stage is the least visible and among the most important for user experience.

Stage 6: Action (Doing the Thing)

The action stage executes the understood intent against the real world. This is where the assistant stops being a language processor and starts being an agent — calling APIs, querying services, controlling devices.

The actions fall into categories:

Device-local: set timers, toggle settings, control smart home devices on the local network. Fast, private, works offline.

First-party services: the vendor’s own offerings — Amazon’s shopping, Apple’s music, Google’s search. Deep integration, smooth experience.

Third-party integrations: Skills, Actions, Shortcuts — the platform layer where developers extend the assistant. Variable quality, variable reliability.

Web fallback: when nothing else matches, search the web and read the answer. The universal backstop.

What’s notable is how much of “assistant intelligence” is actually integration breadth. An assistant that controls 100,000 smart home devices feels smarter than one controlling 1,000 — not because its AI is better, but because its action space is larger. Capability is as much about plumbing as about models. This connects to the broader question of how AI agents make decisions — voice assistants are among the most deployed agents in existence.

Stage 7: Response Generation (Meaning Back to Sound)

The final stage converts the action’s result into something you can hear. Two sub-steps:

Response planning decides what to say — the content, the phrasing, the level of detail. “It’s 72 and sunny in Chicago” versus a five-day forecast: the system chooses based on what you asked and how much you want to hear. Good response planning is concise; bad response planning reads you a Wikipedia article.

Speech synthesis converts the text to audio using neural text-to-speech — generating natural-sounding speech with appropriate prosody and intonation. Modern TTS is essentially indistinguishable from human speech for short utterances, which is both impressive and slightly unnerving. The full story is in my guide on how text-to-speech works.

The response stage closes the loop: sound became text became meaning became action became sound. A second of your life, seven stages of AI.

Where the Pipeline Runs: Device, Edge, Cloud

Each stage can run in different places, and the placement defines the product:

On-device (phone, speaker): wake-word detection always; increasingly ASR and NLU (Apple’s approach). Benefits: privacy, latency, offline use. Costs: model size limits, slower updates.

Cloud (data centers): the traditional home of ASR, NLU, and action execution (Alexa’s approach). Benefits: enormous models, instant updates, vast compute. Costs: latency, connectivity dependence, privacy exposure.

Edge (local network, e.g., a home hub): emerging middle ground — more compute than the device, more privacy than the cloud.

The industry trend is toward moving more stages on-device as phone chips get more powerful. But the cloud retains structural advantages for the hardest problems — the biggest models still live in data centers, and probably always will.

How Errors Cascade Through the Pipeline

Here’s the pipeline’s dirty secret: errors compound. Each stage’s output is the next stage’s input, so a small error early becomes a big failure late.

Mangled audio (stage 1) → wrong transcript (stage 3) → wrong intent (stage 4) → wrong action (stage 6) → confident wrong answer (stage 7). The assistant doesn’t know it’s wrong — each stage processed its input faithfully. The failure is systemic, not local.

This is why pinpointing voice assistant failures requires stage thinking. “It heard me wrong” (ASR error) needs different fixes than “it understood me wrong” (NLU error) or “it did the wrong thing” (action error). Users experience one failure — “it’s broken” — but engineers see seven possible culprits.

The cascade also explains why end-to-end optimization matters more than stage-level optimization. A 1% improvement in ASR accuracy can produce a 3% improvement in task success, because it prevents downstream cascades. The pipeline is only as strong as its weakest stage — and the weakest stage is usually the one getting the messiest input.

The Pipeline Is Collapsing (in a Good Way)

The seven-stage pipeline I described is the classical architecture. The frontier is collapsing it.

End-to-end spoken language models — systems that go directly from audio to audio, without explicit text in the middle — are emerging. They learn the whole pipeline jointly, which eliminates cascade errors and enables capabilities the staged approach can’t match: understanding tone and emotion, handling interruptions naturally, responding with appropriate prosody.

The tradeoff is interpretability and control. The staged pipeline lets engineers inspect and fix each stage independently. An end-to-end model is a black box — when it fails, you can’t tell which “stage” broke because there are no stages. The industry is navigating this tradeoff in real time.

My prediction: the pipeline doesn’t disappear, it becomes fractal. End-to-end models handle the fluid conversational parts; staged components handle the high-stakes actions where auditability matters.

Frequently Asked Questions