Latency is the product: why response time decides if AI calls work
Why a few hundred milliseconds separates an AI call that feels natural from one that feels broken, and where the time actually goes.
Two AI agents can use the same model, the same voice and the same instructions, and one will feel like a conversation while the other feels like a bad video call. The difference is usually not intelligence. It is a few hundred milliseconds.
The number that matters
In natural conversation, the gap between one person finishing and the next starting is short — commonly around 200 milliseconds, and people begin to register a pause as meaningful somewhere past half a second. By about a second, the silence is doing work: it reads as hesitation, confusion, or a dropped line.
This is why callers say an agent felt "robotic" when the wording was fine. They are not describing the words. They are describing the timing.
Where the time goes
| Stage | What happens | Typical cost |
|---|---|---|
| Endpointing | Deciding the caller has finished | 100–300ms |
| Transcription | Audio to text | 100–200ms |
| Model | Generating the reply | 300–900ms |
| Speech | Text to audio, first byte | 100–300ms |
| Network | Carrier and internet transit | 50–150ms |
Run those sequentially and you are between 650ms and nearly two seconds. Sequential is what a naive implementation does, and it is why prototypes feel wrong.
How the gap gets closed
- Stream everything. Transcribe while the caller is still speaking. Start generating on a partial transcript. Start speaking on the first sentence rather than the last.
- Speculate. Begin generating before endpointing is certain, and discard the work if the caller continues. Wasted tokens are cheaper than dead air.
- Keep prompts short. Input length costs time as well as money. A prompt with the entire knowledge base pasted in is slow on every turn.
- Choose models for latency on the call path. A slightly less capable model that answers in 300ms usually produces a better call than a stronger one at 900ms.
- Put compute near the carrier. Cross-region hops are avoidable milliseconds.
Barge-in is part of the same problem
Real conversations interrupt. When a caller starts talking over the agent, it must stop within roughly 200ms and discard what it was about to say. An agent that talks over people is worse than one that is slightly slow, because it feels like it is not listening — which, at that moment, it is not.
Consistency beats a good average
A median of 500ms with occasional four-second stalls is worse than a steady 700ms. Humans adapt quickly to a consistent rhythm and are thrown by a variable one. When you evaluate, look at the worst calls rather than the average — the distribution is the experience.
This is also the strongest argument against building your own: closing this gap, and keeping it closed across provider changes, is most of the engineering. See build or buy for the rest of that calculation.