Research note
Code-mixing is the default, not the edge case
Systems built for one language per conversation break on the first real Indian sales call. What it costs to treat mixed speech as the normal input.
Almost every speech stack you can buy assumes a conversation has a language. You set it at the start of the session, or you run a detector on the first few seconds and then commit. It is a clean abstraction, and on an Indian sales call it is wrong within about two sentences.
A real utterance from a customer looking at a property listing sounds closer to this:
Haan main dekh raha hoon, but budget thoda tight hai, can you send the floor plan on WhatsApp?
Three switches in one sentence, none of them at a clause boundary you could detect cheaply. This is not a speaker being unusual. In urban Indian business conversation this is the register, the unmarked and default way people talk. A system that forces a choice between Hindi and English is not handling an edge case badly, it is mishandling the common case.
What breaks first
The failures cascade in a specific order, and it is worth knowing the order because it tells you where to spend effort.
- Recognition. A model decoding with a Hindi language model will render "budget" as a phonetically similar Hindi word. It is not that it fails to transcribe. It confidently transcribes something else, which is much worse for everything downstream.
- Script. Even with correct recognition, you now have to decide what to write down. Romanised Hindi, Devanagari, or a mix? Sales reps type romanised into CRM notes. Choosing Devanagari for the agent's transcript means the rep's search never finds the agent's call.
- Intent. Entity extraction trained on monolingual text misses numbers and amounts that arrive in the other language. "Two lakh" and "do lakh" are the same number and rarely the same token.
- Synthesis. Reading an English word with Hindi phonology, or the reverse, is instantly noticeable. It is the single thing that most reliably makes callers say "this is a robot".
Two approaches, both with a real cost
The obvious fix is to route per turn: detect the dominant language of the utterance, then hand it to the matching model. This is cheap and it is what we started with. It also loses precisely the thing that matters, because the dominant language of the sentence above is not a useful description of it. You get a transcript that is correct in one language and mangled in the other, and the mangled half is usually where the entities live.
The other approach is to route per token, or to use a model that was trained on mixed input in the first place so no routing is needed. This is better output and it costs latency, and as we wrote in the turn budget note, we have about 130 ms for transcription and no slack to give away.
Where we landed, for now: a single recognition pass with a vocabulary that covers both languages and the romanised forms, rather than a routing layer over two monolingual models. It costs a little accuracy on purely monolingual calls. It costs far less on the calls we actually carry.
The script decision is a product decision
This one surprised us, because it looks like a technical detail and it is not. When a Voiceplix call ends, the outcome gets written back to the record in Astrosia CRM. A human rep will read that note, search it, and act on it.
Reps type romanised. They search romanised. If we write Devanagari, we have produced a transcript that is arguably more correct and practically invisible to the person who needs it. So we normalise to romanised Latin script for the written record while keeping the original for the audio-aligned transcript. Correctness lost the argument to findability, and we think that was right.
Numbers deserve their own path
The single highest-value entity on a sales call is an amount, and amounts are where code-mixing does the most damage. Indian numbering, lakh and crore, mixes with English digits freely, and speakers switch mid-number: "twenty-five lakh", "paanch crore", "two point five lakh".
We stopped trying to solve this inside the language model and gave numbers a dedicated normalisation pass over the transcript, with the Indian numbering system as a first-class citizen rather than a locale setting. It is unglamorous, rule-heavy code. It is also the change that moved the needle most on whether the outcome written back to the CRM was usable.
What is still open
Switching within a single word, an English stem with a Hindi inflection, happens constantly in speech and still degrades us. So does synthesis: getting the agent to read a mixed sentence with the right phonology for each word, without an audible seam at the switch, is not solved. Both are on the open problems list, and both are the kind of thing where the honest answer today is that we are better than we were and not where we want to be.