Most localization pipelines assume one source language per recording. A producer tags a file as Hindi, the recognition stage loads a Hindi model, and every downstream step inherits that decision. Then a speaker says, "the deployment window is tight, humein logistics ke saath align karna padega," and the transcript comes back with the English words transliterated into Devanagari, duplicated, or dropped. The translator, who never heard the audio, renders it faithfully.
Code-switching in video translation is not an edge case. Bilingual speakers switch because a term has no settled equivalent, because a quotation is only valid in its original form, because a professional register pulls in the English noun, or because they are more fluent in one language for one subject. The switch carries information, and a pipeline that treats it as noise produces output that is technically translated and practically wrong.
The models are rarely the obstacle; most handle two languages when told to. Nobody tells them, because the metadata says one language and mixed speech is rarely distributed evenly through a file. Ten minutes of monolingual interview followed by ninety seconds of dense switching is still tagged as a monolingual asset.
Why code-switching in video translation breaks single-language ASR
What the decoder does when the language changes
Speech recognition combines an acoustic score, which measures how well the audio matches a sequence of phones, with a language score, which measures how likely that token sequence is in the expected language. That second score is where code-switching does its damage. When a Hindi model hears "deadline," the probability mass assigned to a Devanagari transliteration far exceeds the mass assigned to the English spelling, so the decoder writes डेडलाइन. The audience gets a Hindi noun where a shared English term belonged, and the intended register is gone.
The acoustic model adds a second failure. Phoneme inventories differ across languages, and a model trained on one maps unfamiliar phones onto its nearest neighbors. English /θ/ and /ð/ collapse onto dental stops in a Hindi-tuned system; French nasal vowels flatten in an Arabic-tuned system; Swahili vowel length distinctions disappear under an English-tuned system. The transcript is phonetically wrong, not merely mislabeled, so correction by inspection cannot always recover the original word.
The damage downstream of the transcript
Errors do not stay in the transcript. Punctuation restoration and sentence segmentation rely on the language model's sense of where clauses end, so confused boundaries become cue boundaries and subtitle timings. Termbase matching fails because the source string is transliterated rather than spelled conventionally, and translation memory lookups miss. Speaker diarization still separates voices, but one speaker who switches languages may be split into two if language is one of its features.
Word error rate adds a measurement problem. The reference transcript must be code-switched the same way the audio is; a reference that normalizes "schedule" to a Hindi equivalent scores a correct English token as an error and a wrong Hindi token as correct.
A worked example
Take a Spanglish line from a product update: "Necesitamos cerrar el rollout antes del Q3, pero el budget todavía está en review." Three English tokens sit inside a Spanish frame, and each behaves differently. "Rollout" is an established borrowing that Spanish-speaking audiences read without friction. "Q3" is a proper noun that stays as-is in nearly every target language. "Review" is the interesting one: it can be translated as revisión, but in this speaker's organization the English word may be the internal term.
An English-only pipeline outputs "Necesitamos cerrar el roll out antes del cute three," or drops the Spanish. A Spanish-only pipeline transliterates all three English tokens into Spanish orthography. Neither output is usable without a human who can hear the audio.
Common patterns in a real library
Hinglish and South Asian mixing
Hindi-English switching follows domain. Technical and commercial nouns arrive in English; verbs, connectors, and emotional content stay in Hindi. The mix is grammatically integrated rather than two languages stapled together: a Hindi verb can carry an English stem with Hindi morphology, so the switch happens inside a word rather than between words. Word-level language identification on such audio is unreliable.
Spanglish in bilingual content
In United States bilingual media, switching often tracks audience rather than topic. A speaker addressing a monolingual English colleague stays in English, then turns to a Spanish-dominant family member and switches mid-sentence. The switch is a social signal, and removing it removes characterization.
Taglish, Arabic-French, and Swahili-English
Taglish resembles Hinglish in structure but differs in default: English supplies the technical layer, Tagalog supplies affect and hedging. Arabic-French switching in North African and Levantine content marks formality in French and intimacy in Arabic. Swahili-English switching in East African broadcast and corporate content tends toward English for institutional vocabulary and Swahili for narrative. The direction of the switch tells you which language is the frame and which is the insert.
What the patterns share
- One language supplies most function words and the other supplies content words, so the switch is asymmetric.
- The switch is topic-bound, so it clusters in time rather than distributing evenly.
- The switch is intentional, and it is invisible in project metadata because the file was tagged before anyone listened.
Code-switching in video translation: intra-sentence versus speaker-level
Intra-sentence switching
A single utterance contains tokens from two languages. Language identification must operate below the utterance level, the acoustic model must survive two phoneme inventories, and translation must decide token by token whether to translate or preserve. Intra-sentence switching also breaks the assumption that a cue has one source language, which changes how you configure a video translation pipeline and route output for review.
Speaker-level switching
Different speakers use different languages, or one speaker uses a different language in clearly separated segments. This is easier because segmentation can carry the language label. The risks are mislabeled segments, abrupt timbre changes when a voice model is reloaded, and inconsistency when the same proper noun is spelled two ways across the file.
Why the handling differs
Speaker-level switching is a routing problem; intra-sentence switching is a token-level decision problem. Conflating them produces the worst of both: segment-level routing that splits a sentence in half, and token-level heuristics applied to a speaker who never mixed languages.
A workable sequence for a mixed asset:
- Detect language at the segment level and flag segments whose confidence is low or whose language differs from the file tag.
- Re-run recognition on flagged segments with a two-language configuration.
- Run word-level language identification on flagged segments only, recording per-token confidence.
- Classify each flagged token using the decision rules below.
- Route anything that fails classification to a bilingual reviewer with a timestamp and a short audio excerpt.
Deciding what the switch means: term, quote, or habit
The lexical-gap test
Ask whether the target language has a settled equivalent a domain expert would recognize. If it does and the speaker still used the other language, the switch is stylistic. If it does not, the switch marks a real gap, and the borrowed term should usually survive translation, perhaps with a gloss on first use.
Register and institutional terms
Corporate titles, product names, compliance vocabulary, and internal jargon stay in the source language because the organization uses them that way. A Tagalog-dominant speaker in a Manila office may say "quarterly business review" in English because the calendar invite does, and translating it into Tagalog can make the line harder to match against the organization's own materials.
Social and identity signals
When a speaker shifts language to address a family member or quote a parent, the switch is character information, and translating it away flattens the scene. Preserving it asks the audience to read a line in a language they do not share, which is a legibility trade-off rather than a translation problem.
Keeping a switch ledger
For any asset with more than a handful of switches, keep a ledger that travels with the project:
- Timestamp range and speaker
- Detected language and the confidence that triggered review
- Classification: term, quotation, register marker, social signal, or unclear
- Decision: preserve, translate, gloss, or flag
- One-line rationale from the person who made the call
- Whether the same switch exists in the target audience's repertoire
The ledger turns a per-cue argument into a reusable rule. When the same presenter appears in a second video, the decisions carry forward.
Subtitle conventions for marking a second language on screen
Italics and when they stop working
Many broadcast style guides italicize foreign-language dialogue. That convention assumes the audience shares a dominant language and the foreign line is an exception. In a genuinely bilingual asset, where a third of the dialogue is in the other language, italics stop signaling "foreign" and start signaling "different," which readers interpret as arbitrary.
The target-language collision problem
If the source mixes Hindi and English and the target is English, the Hindi needs translation and the English insert does not. Translate both and the audience loses the fact that the speaker switched; translate neither and comprehension fails. The usual resolution is to translate the non-target language and leave the target-language insert untouched, so the switch stays visible. A subtitle translation workflow should expose this as a per-project setting rather than hardcode it.
Dashes, tags, and speaker attribution
Dialogue dashes assume one language per cue. Where a cue spans a switch, some conventions insert a bracketed language tag and others use a dash plus italics on the switched portion. Tags survive small screens and auto-generated captions better than formatting, since italics may not render. Pick one and apply it across the series.
When not to translate
If a scene is deliberately opaque to a character who does not speak the second language, translating the line destroys the dramatic function. Subtitle it anyway for accessibility or mark it as untranslated, but the decision belongs to the producer. Log it in the ledger.
Dubbing decisions: preserve, normalize, or flag for a human
Preserve the switch
Preservation works when the target audience plausibly shares the second language. A Spanish dub of an English-Spanish interview can leave the English lines in English, because the switch is part of what the audience came for. This option lives or dies on delivery: leave English lines to a voice actor with a heavy accent and the result reads as a mistake rather than a choice.
Normalize to one language
Normalization works when the switch is habitual rather than meaningful. If a speaker uses "budget" and "presupuesto" interchangeably in the same paragraph, translating both to presupuesto loses nothing. It also simplifies timing, because one target language produces more predictable syllable counts for lip synchronization.
Flag for human review
Flag when the switch is a plot point, a quotation, a legal statement, or a term of art, and when the speaker's accent in the second language is itself characterization. Automated systems cannot reliably distinguish a speaker who is fluent from one who is performing fluency, and that distinction changes the correct treatment in a dubbing pass.
Voice casting and voice preservation
A dub that carries two languages is a casting decision: either one voice actor handles both convincingly, or two actors split the part with a risk of audible discontinuity. Speaker diarization establishes how many distinct voices exist before you assign any of them. Where the original speaker's identity matters, voice cloning preserves that voice across target languages, provided the speaker has authorized the use; speech generation for the target lines should follow the same authorization record as the source recording.
Segmentation and timing problems in mixed-language audio
Word boundary drift
Recognition errors near a switch point shift word boundaries. A segment that should start on a conjunction starts half a word later, and because cues are built from those boundaries, one bad boundary propagates through the paragraph. When you re-run recognition on a flagged segment, re-derive cue timing for that segment and the following one rather than only the flagged one.
Reading speed across a two-language cue
A cue containing two languages runs longer than a monolingual cue with the same content, because the switch consumes screen space, especially with a language tag. A line that reads comfortably in one language can exceed a comfortable rate once a tag or gloss is added. Recheck characters per second on every cue carrying a switch marker, and split rather than compress.
Numbers, names, and punctuation
Numbers spoken in one language and written in another are a common mismatch: the speaker says the number in Hindi, the surrounding text is Spanish, and numeral formatting differs by locale. Decide once whether numbers follow the target locale or the source. Names behave the same way when they have a conventional transliteration in one language and not the other.
Test clips and a review workflow for ambiguous switches
What the clip set must contain
Build the evaluation set from your own library rather than public corpora, because your failure modes come from your speakers. Fifteen to twenty clips is enough if they cover the right axes:
- Monolingual audio in each language your library uses, as a control
- Intra-sentence switching with short inserts of one or two words
- Intra-sentence switching with inserts that span a clause
- Speaker-level switching between two speakers, and within one speaker across topic changes
- Switching under noise, music, or overlapping dialogue, where the switch point is often the noisiest part of the clip
- At least one quotation and one term of art
Thresholds and routing
Define an explicit rule for usable output. A workable start: a token passes automatically when the detected language is confident and the classification is unambiguous, and is flagged when confidence is low, when classification is "unclear," or when the token is a proper noun or numeral that exists in both languages. Tune the threshold against your clip set by measuring how many flagged tokens route to review and how many of those reviews change the output; a threshold that sends everything to review is as useless as one that sends nothing.
Bilingual reviewer instructions
Give reviewers three things: the audio with the switch timestamp marked, the current output, and an instruction sheet asking one question per flagged token, namely whether the speaker switched languages and what the switch does. Reviewers answer in a controlled vocabulary drawn from the ledger's classification list, so their answers feed back into the termbase and into the next project's rules. Teams wiring this into an existing system can exchange review metadata through the API, with the schema and field definitions in the documentation, keeping timestamps, speaker labels, and review status attached to the asset instead of a separate file.
Frequently asked questions
Can automated systems detect code-switching reliably?
Segment-level detection is reliable when the language changes for more than a few seconds and the audio is clean. Word-level detection inside an utterance is far less reliable, particularly for single-word inserts and for languages that share loanwords. Treat it as a candidate generator that requires review.
Should subtitles translate a switched line or leave it in the source language?
Translate it if the target audience cannot be expected to understand it, and leave it if the switch is the point. Most productions translate for accessibility and preserve the switch through formatting or a language tag.
Does dubbing always remove the original switch?
No. A dub can preserve the second language when the target audience shares it, normalize both into one when the switch is habitual, or flag the line for a human decision when the switch is meaningful. The choice is editorial, not technical.
What is the biggest practical cost of ignoring code-switching?
Rework. Errors introduced at recognition compound through translation, timing, and quality control. A switch caught at the transcript stage costs one correction; the same switch caught at subtitle review costs a re-translation, a re-timing pass, and a new review cycle.
How do you handle a speaker who switches to quote someone else?
Treat the quotation as its own unit. Preserve it if the audience is meant to recognize the original wording, and translate it with a quotation marker if the content matters more than the form. Record the decision in the ledger so a repeated quotation is handled consistently.
Do you need a bilingual reviewer for every project?
Only where switches are frequent or carry meaning. A single reviewer who speaks both source languages can cover a series, and their ledger decisions reduce the volume of future reviews.
How many languages should a recognition pass be configured for?
Two covers most real switching. Three-language intra-sentence switching exists but is rare, and adding a third language increases false positives on monolingual audio more than it improves recall on mixed passages.
Conclusion
The practical first step is not a model change. It is a listening pass. Pull a sample from each series in your library, listen for switches, and note which pairing dominates and how the switch behaves. That tells you whether two-language recognition needs to run on every asset or only on a flagged subset, and which reviewer you need to hire.
From there, set your defaults deliberately. For subtitles, decide whether you translate switched lines or preserve them, and pick a marking convention you can apply consistently. For dubbing, decide whether you preserve, normalize, or flag, and write the rule down so the outcome does not depend on who is on shift.
The last step is the loop. Every ambiguous switch a reviewer classifies becomes a rule, and every rule shrinks the next project's review queue. Teams that run this loop end up with a termbase that reflects how their speakers actually talk, which is the only asset that makes mixed-language production sustainable. If you are starting from scratch, a short conversation about how your audio and your review workflow should connect is faster than rebuilding the pipeline twice; get in touch before the next series goes into production.
