Research note
The turn budget: what it takes for a phone call to feel human
A conversational turn has about 800 milliseconds before the caller notices. Here is how we spend them, and which stage we lose first when the line is bad.
People are unforgiving about silence on a phone call. In face-to-face conversation the gap between two speakers averages around a fifth of a second, and a listener starts predicting the end of your sentence long before you get there. On a phone line the visual cues are gone, so the only signal the caller has that the other side is still present is that the other side answers quickly.
That is the whole design constraint behind Voiceplix. Not accuracy, not naturalness of voice. Those matter, but they are recoverable. Latency is not. If the agent takes too long, the caller assumes the line dropped, says "hello? hello?", and now two speakers are talking at once. There is no graceful recovery from that in an automated system.
Where the milliseconds go
We design a single turn, from the caller stopping to the agent starting, to a budget of roughly 800 ms. That number is a target, not a measured average across every call we carry; real calls vary with the carrier, the handset, and the network the caller is standing on. But it is the number every component gets designed against, and each stage has an allowance.
| Stage | Allowance | What is actually happening |
|---|---|---|
| Carrier leg | ~120 ms | Audio arriving over the SIP trunk, jitter buffer settled enough to be useful |
| Endpointing | ~140 ms | Deciding the caller has genuinely finished, not just paused |
| Transcription | ~130 ms | Streaming recognition catching up to the final words |
| Reasoning | ~250 ms | Intent, any retrieval, next action, first token out |
| Speech | ~160 ms | First audio frame synthesised and pushed back down the leg |
The carrier leg is the part we do not control. It is bought, not built. Voiceplix rides licensed carrier partners, and that leg is also the ₹0.65/min that sits underneath every published all-in rate. What that means practically is that roughly fifteen percent of our budget is spent before our code runs at all, and it is the portion most likely to degrade without warning.
Endpointing is the expensive problem
The naive approach to deciding when a caller has finished is a silence timer: no energy above a threshold for N milliseconds, therefore the turn is over. This works beautifully in a demo recorded on a laptop microphone in a quiet room. It falls apart on a call from a footpath in Bengaluru at 6pm.
Two failure modes, and they pull in opposite directions:
- Cutting the caller off. Someone says a phone number, pauses to read the next digits off a screen, and the agent has already started responding. The caller has to start over, and the transcript is garbage.
- Waiting too long. Background noise, whether traffic, a television, or another conversation, keeps the energy above threshold, so the timer never fires. The agent sits there while the caller waits.
Every fix for the first makes the second worse. Raise the silence threshold and you cut people off more; lower it and you hang on noise. A fixed threshold is the wrong shape of solution, because the thing that varies is not the caller's speech, it is the floor underneath it.
Where we are: we adapt the threshold to the measured noise floor of the specific call in its first seconds, rather than shipping one global number. It is a clear improvement and it is not sufficient. A caller thinking mid-sentence and a caller who has finished still look similar to us, and that remains on our open problems list.
Reasoning gets the biggest slice, and should
250 ms is the largest single allowance, and the temptation is always to spend it on a better answer: more retrieval, a bigger model, a second pass to check the first. We have learned to resist almost all of that, for a reason that took a while to internalise.
The metric that matters is not total time to a complete response. It is time to first audio frame. Once the agent starts speaking, the caller is occupied, and the rest of the response can be generated while they listen. So the entire pipeline is built to stream: the model emits tokens, we chunk them at clause boundaries, synthesis starts on the first chunk, and audio goes out while the model is still writing the end of the sentence.
This changes what counts as an expensive operation. A retrieval step that adds 200 ms before the first token is very expensive. The same retrieval running in parallel with the opening clause of a response is nearly free. Most of our optimisation work is not making things faster. It is moving work to the other side of the first audio frame.
Barge-in changes the arithmetic
Callers interrupt. They interrupt constantly, and usually because they already know what the agent is about to say. Handling that well means the agent has to stop speaking almost instantly, and the cost of stopping is not symmetrical with the cost of starting.
When the agent is speaking, its own audio is on the line. Detecting that the caller has started talking underneath it requires separating the two, and doing so fast enough that the agent goes quiet within a couple of hundred milliseconds. Get it wrong in one direction and the agent talks over the caller; get it wrong in the other and the agent stops speaking because of an echo of itself.
The practical consequence: we keep synthesised audio buffered in small chunks rather than handing a long clip to the carrier. A large buffer is efficient and makes barge-in feel broken, because the caller keeps hearing the agent for a second after it has logically stopped.
What we would tell someone starting on this
- Measure to first audio frame, not to response complete. They diverge quickly once you stream, and only one of them is what the caller experiences.
- Budget the carrier leg explicitly. It is real time, you do not control it, and leaving it out of the budget means the first bad network day looks like a code regression.
- Test on the worst line you can find, not the best. Clean-audio benchmarks hide exactly the failure that will define your production experience.
- Treat barge-in as a first-class path, not an edge case bolted on later. It reaches into buffering, synthesis, and state management, and it is very hard to retrofit.
None of this is finished. The endpointing work in particular is where most of our current effort goes, and we will write about it again when we have something more definitive than "adapt to the noise floor and keep testing".