Should you build or buy an AI phone agent?
A frank breakdown of what building your own actually involves, and the small number of situations where it is the right answer.
Wiring a speech model to a language model to a phone line looks like a weekend project. The demo genuinely is one. The distance between that demo and something you would put in front of a paying customer is where the year goes.
What the weekend gets you
A working prototype. Audio in, transcription, a model reply, speech out, over a carrier trunk. It will hold a simple conversation and it will feel like the problem is solved.
What it does not get you
The prototype has a two to four second gap between the caller finishing and the agent starting. Humans tolerate roughly 500 to 800 milliseconds before a pause reads as awkward. Closing that gap is not an optimisation pass — it is streaming at every layer, partial-transcript speculation, canceling in-flight generation when the caller speaks again, and it is most of the engineering.
Then there is everything that is not latency:
- Barge-in. The caller interrupts. You must stop speaking within about 200ms and discard what you were going to say.
- Endpointing. Deciding whether a pause means 'your turn' or 'I am thinking'. Get it wrong in one direction and you interrupt customers; in the other, dead air.
- Failover. A provider degrades mid-call. You need a second one warm, and a way to switch without the caller hearing it.
- Telephony reality. DTMF, transfers, voicemail detection, codecs, jitter, one-way audio.
- Compliance. Recording disclosure, consent capture, do-not-call suppression, quiet hours, an audit trail.
- Observability. Per-call latency, token cost, failover events. Without these you cannot tell a bad week from a bad model.
The cost nobody models
Ongoing maintenance is the term that sinks build decisions. Model providers deprecate versions. Prices change. A prompt that behaved well on one model version behaves differently on the next. Carrier APIs change. Somebody has to own that permanently, and that person is now doing this instead of the thing your business actually sells.
When building is genuinely right
- The agent is your product, not support for your product. Then this is core engineering and should not be outsourced.
- Volume is very large. Above a certain minute count, per-minute margin exceeds a team's salary, and the arithmetic flips.
- A hard data-residency or regulatory constraint that no vendor will meet contractually.
- A genuinely unusual conversation shape that no product models — something no configuration surface can express.
Note what is not on that list: wanting control, wanting to save money at low volume, or having engineers available. Those are the three reasons most build decisions are made, and they are the three that most often reverse a year later.
The middle path
Buy the conversation layer, build the integrations. Let a product handle latency, barge-in, failover and compliance. Spend your engineering on the part that is specific to you: your scheduling logic, your pricing rules, your CRM.
That is where the defensible value is anyway. Nobody wins a market because their speech pipeline was slightly better.
If you are evaluating rather than building, the readiness test and why latency decides everything are the two worth reading next.