Callers do not report latency. They just decide the bot is stupid, talk over it, or hang up. That is why this metric deserves more attention than voice quality, script wording, or model choice — you can fix all three and still lose the call to a pause. This guide covers what the numbers should be, where the milliseconds actually go, and which fixes are worth your engineering hours.
The 200ms Benchmark You Are Being Judged Against
In human conversation, the gap between one person finishing and the other starting is remarkably tight. Conversation analysis research has found that the typical turn-taking gap is around 200 milliseconds across many languages. That is not a target most AI voice systems will consistently achieve. It is the benchmark your caller's brain unconsciously uses when judging whether a conversation feels natural.
This is why latency often reads as intelligence. A person who takes two seconds to answer a simple question sounds confused or distracted. AI voice agents are judged the same way. The words can be perfect, but if they arrive too late, the caller may already feel the interaction is unnatural.
What Counts as Fast in 2026?
End-to-end latency | How it lands | Verdict |
Under 800ms | Feels like a conversation | Target |
800–1,200ms | Slightly deliberate, still fine on business calls | Acceptable |
1,200–1,500ms | Noticeable pause; callers start repeating themselves | Fix it |
Over 1,500ms | Obvious machine; talk-over and hang-ups climb | Broken |
Two clarifications that matter. First, measure from the moment the caller stops speaking to the first audible audio — not your LLM round-trip. Second, report P95, not the median. An agent with a 700ms median and a 2.5-second P95 is not a fast agent; it is a fast agent that periodically humiliates you, and the tail is what callers remember.
Where the Milliseconds Actually Go
A cascaded voice pipeline has five places to lose time. Rough per-stage figures reported across the industry in 2026:
Stage | What it does | Typical cost |
Endpoint detection (VAD) | Decides the caller has finished talking | Often the biggest and least-measured chunk |
STT | Transcribes speech to text | ~150ms+ to first tokens, streaming |
LLM | Decides what to say; runs tool calls | ~250–500ms to first token |
TTS | Generates audio from text | ~150–300ms to first audio (TTFA) |
Network / telephony | Carrier hops, region distance, vendor boundaries | Silent tax; grows with every hop |
Add those up honestly and the 800ms budget is tight but reachable — provided nothing waits for anything else to finish. The moment one stage blocks on the previous stage completing, the budget is gone. If any of these components are unfamiliar, our primer on what speech AI is explains the ASR, LLM, and TTS layers.
Why Does My Voice Agent Feel Slow When the Numbers Look Fine?
Because you are almost certainly measuring the wrong thing. Three culprits, in the order we usually find them.
Endpoint detection is eating your budget. Every STT provider ships conservative defaults — they wait, in case the caller has not finished, because cutting someone off is more annoying than dead air. On phone calls that default is too cautious. Silence thresholds tuned for dictation add hundreds of milliseconds to every single turn before your pipeline has done any work at all. This is the first place to look and the last place most teams check.
You are buffering between stages. Each component may be fast in isolation while the pipeline waits for complete outputs before passing them on. STT waits for the final transcript, the LLM waits to finish the sentence, TTS waits for the full text. Three fast components, stitched serially, produce a slow agent.
Your median is hiding your tail. See above. Instrument P95 per stage, per turn, in production — not in a demo, on your laptop, next to the data centre.
The Fixes, Ranked by Payoff
In rough order of milliseconds saved per hour of engineering:
1. Tune endpoint detection. Lower silence thresholds for telephony, or move to semantic turn detection that recognises a completed thought instead of just absence of sound. Biggest single win, usually.
2. Stream everything, end to end. STT emits tokens as it hears them; the LLM streams tokens as it generates; TTS starts synthesising the first clause while the model is still writing the rest. Nothing waits for a full output.
3. Fire speculatively on interim transcripts. When the interim transcript looks grammatically complete, start the LLM call. If the caller keeps talking, cancel it. Reported savings are in the 300–500ms range 200
4. Right-size the model. A smaller, faster model with low time-to-first-token beats a large one for most phone tasks. Route the genuinely hard turns to the bigger model deliberately, not by default.
5. Pick TTS on time-to-first-audio, not on demo-reel quality. A voice that starts speaking in ~150ms feels dramatically more alive than a prettier one that starts at 400ms.
6. Collapse the hops. Co-locate components; cut vendor boundaries. See below.
Perceived Latency: The Cheapest 300ms You Will Ever Buy
There is actual latency and there is felt latency, and only one of them is on your dashboard. Humans do not sit in silence while thinking — they say “right,” “let me check,” “mm-hm.” A voice agent that does the same buys real processing time without the caller experiencing a gap.
Two mechanics do most of the work.
Filler audio: the agent acknowledges immediately (“Sure — one moment”) while the tool call runs behind it. Backchannelling: small acknowledgements during a long caller utterance, so the line never goes dead. Neither reduces a single millisecond of measured latency. Both change the answer to “did that feel slow?” — which is the only question that matters.
Use them honestly, though. Filler that fires on every turn becomes a verbal tic, and callers notice a bot that says “great question!” four times in ninety seconds.
Latency Tolerance Is Not Uniform
This is the nuance the benchmark tables miss: the same 1.2 seconds is invisible in one place and fatal in another.
Moment in the call | Tolerance | Why |
Opening greeting | Very low | Dead air on pickup reads as a dropped call — people hang up |
Simple factual turn | Low | “Yes, we’re open till six” should be instant, or the agent sounds unsure |
After “let me check that” | High | You have announced a wait; a two-second pause is now expected |
Mid-sentence barge-in | Near zero | If the agent keeps talking over an interruption, the call is effectively over |
So do not spend a sprint shaving 200ms off a database lookup the agent has already excused. Spend it on the greeting and on barge-in handling, where the caller has no patience at all. Slow greetings are also a direct contributor to call abandonment.
How to Measure It Properly
• Measure in production, over real telephony. Lab numbers over WebRTC on office wifi are marketing, not diagnostics.
• Measure from caller-speech-end to first audio byte. Anything else flatters you.
• Report P50 and P95 per stage. The gap between them is your real problem.
• Segment by region. A single global deployment means someone is always talking to a distant data centre.
• Watch talk-over and hang-up rates as behavioural proxies — they move before your latency dashboard does.
Conclusion
Latency is the difference between a voice agent that sounds capable and one that sounds broken, and the bar is set by human conversation, not by your architecture. Aim for sub-800ms end to end, report P95 rather than the median, and diagnose before you optimise — the thief is usually endpoint detection or serial buffering, not the model everyone wants to blame. Then buy back what physics will not give you: stream every stage, fire speculatively, choose TTS on time-to-first-audio, cut vendor hops, and use filler audio so the caller never hears the gap you could not close. Get under a second and the conversation stops being about the technology, which is the entire point.
FAQs
What is AI voice agent latency?
It is the delay between a caller finishing their sentence and the agent starting to speak. It accumulates across endpoint detection, speech-to-text, the language model, text-to-speech, and network hops. It is the main driver of whether a voice agent feels natural or obviously automated.
What is a good latency for an AI voice agent?
Under roughly 800 milliseconds end to end feels like a natural conversation. Between 800 and 1,200ms is acceptable on business calls. Past about 1,500ms the pause is obvious and callers start talking over the agent or hanging up. Human conversation turns over in around 200ms, which is the instinctive benchmark.
Why is my AI voice agent slow?
Usually endpoint detection: default silence thresholds wait to be sure the caller has finished, adding hundreds of milliseconds per turn. The other common causes are buffering between stages instead of streaming, and vendor hops in a stitched STT/LLM/TTS/carrier stack. The language model is rarely the main culprit.
How do you reduce voice agent latency?
Tune endpoint detection or use semantic turn detection; stream STT, LLM, and TTS end to end so nothing waits for a complete output; fire speculative LLM calls on interim transcripts; use a smaller fast model with low time-to-first-token; choose TTS on time-to-first-audio; and reduce vendor and network hops.
What is the difference between actual and perceived latency?
Actual latency is measured milliseconds; perceived latency is whether the caller experiences a gap. Filler audio (“one moment”) and backchannelling let the agent respond instantly while processing continues, cutting perceived delay without changing the measured number.
