Multilingual Embedding Issues

The Problem

Embedding models perform poorly across languages—English queries don't match non-English documents, translation loses meaning, and cross-lingual retrieval fails.

Symptoms

• ❌ Spanish query returns no results despite Spanish docs existing

• ❌ English query doesn't match French equivalent document

• ❌ Chinese characters embedded as gibberish

• ❌ Must maintain separate indexes per language

• ❌ Machine translation degrades quality

Real-World Example

Knowledge base contains:
→ 500 English docs
→ 200 Spanish docs
→ 100 French docs

User query (Spanish): "¿Cómo autenticar API?"
Translation: "How to authenticate API?"

Embedding model (English-only):
→ Embeds Spanish as unknown tokens
→ Poor semantic representation
→ Returns English docs (wrong language)
→ Misses Spanish "Guía de Autenticación" (perfect match!)

Result: User gets English docs they can't read

---

Deep Technical Analysis

Monolingual Model Limitations

English-trained models fail on other languages:

Vocabulary Coverage:

English tokenizer:
→ Trained on English text
→ Vocabulary: 50K English words/subwords

Spanish text: "autenticación"
→ Not in English vocabulary
→ Tokenized as: ["aut", "##ent", "##ic", "##aci", "##ón"]
→ 5 unknown subword pieces

vs English: "authentication"
→ Single known token
→ Proper semantic representation

Spanish embedding degraded by tokenization issues

Semantic Space Misalignment:

English model learns:
→ "dog" ≈ "puppy" ≈ "canine"

Doesn't learn:
→ "dog" ≈ "perro" (Spanish)
→ "dog" ≈ "chien" (French)
→ "dog" ≈ "犬" (Japanese)

Cross-lingual relationships missing
→ Cannot match concepts across languages

Translation-Based Approaches

Translating before embedding:

Query Translation:

Approach:
1. Detect query language
2. Translate to English (if not English)
3. Embed translated query
4. Search English-embedded docs

Problems:
→ Translation errors compound
→ "Bank" → "Banco" (financial) or "Orilla" (river)?
→ Context lost in translation
→ Idioms don't translate well
→ "It's raining cats and dogs" → ?

Document Translation:

Approach:
1. Translate all docs to English
2. Embed English versions
3. Store original + translation

Problems:
→ Expensive (translate 1000s of docs)
→ Translation quality varies
→ Loses original phrasing/nuance
→ Technical terms mistranslated
→ Updates require re-translation

The Round-Trip Problem:

Original Spanish: "Autenticación de dos factores"
→ Translate to English: "Two-factor authentication"
→ Embed English version
→ User queries in Spanish: "autenticación 2FA"
→ Translate to English: "2FA authentication"
→ Search → Match!

But:
Original Spanish: "Reiniciar contraseña"
→ Translate: "Restart password" (wrong!)
→ Should be: "Reset password"
→ User query: "reset password"
→ Translate to Spanish: "restablecer contraseña"
→ Embed → NO MATCH (reiniciar ≠ restablecer)

Translation errors break retrieval

Multilingual Embedding Models

Models trained on multiple languages:

Multilingual BERT (mBERT):

Trained on:
→ 104 languages simultaneously
→ Shared vocabulary across languages
→ Cross-lingual alignment

Benefits:
→ "dog" and "perro" have similar embeddings
→ Can match across languages

Limitations:
→ Lower quality than monolingual models
→ Diluted by 104 languages (each gets less attention)
→ Still biased toward high-resource languages (English)

Language-Specific Performance:

English: 92% accuracy (high-resource)
Spanish: 85% accuracy (medium-resource)
Vietnamese: 72% accuracy (low-resource)
Swahili: 58% accuracy (very low-resource)

Quality degrades for rare languages
→ Less training data available
→ Poorer representations

Code-Switching and Mixed Content

Documents mix languages:

Within-Document Language Mixing:

English doc with Spanish terms:
"Configure the autenticación de usuario in settings."

Or:
Technical doc with English API terms:
"Pour configurer l'API, utilisez authenticate() method."

Single language model struggles:
→ English model: "autenticación" tokenized badly
→ French model: "authenticate()" tokenized badly

Need model that handles mixed content

The Technical Term Problem:

Universal technical vocabulary:
→ "API", "database", "OAuth", "GitHub"
→ Used across all languages
→ Pronunciation may vary

French doc: "Utiliser l'API OAuth avec GitHub"
Spanish doc: "Usar la API OAuth con GitHub"
English doc: "Using the OAuth API with GitHub"

Technical terms should align across languages
→ But monolingual models don't ensure this

Character Encoding Issues

Non-Latin scripts have encoding problems:

Unicode Normalization:

Same character, different representations:
→ "é" = U+00E9 (single character)
→ "é" = U+0065 + U+0301 (e + combining acute)

Visually identical, different bytes
→ Different tokens
→ Different embeddings
→ Search fails

Must normalize before embedding:
→ NFC (composed) vs NFD (decomposed)
→ Consistent encoding required

Right-to-Left Languages:

Arabic, Hebrew:
→ Text flows right-to-left
→ Rendering direction
→ But stored left-to-right in memory

Embedding model expectations:
→ Trained on left-to-right text
→ May not handle RTL properly
→ Bidirectional text (mixed LTR/RTL) even worse

CJK (Chinese, Japanese, Korean):

No spaces between words:
→ "我喜欢编程" (Chinese: "I like programming")
→ 5 characters, 0 spaces

English tokenizer assumes spaces:
→ Treats each character separately
→ Loses word-level semantics

Need proper CJK word segmentation:
→ "我 喜欢 编程" (I / like / programming)
→ Proper tokenization

Language Detection Challenges

Determining document/query language:

Automatic Detection:

Tools: langdetect, langid, fastText

Short text problems:
→ "API key" (English or universal?)
→ "OK" (English, Spanish, many others)
→ "Taxi" (English, Spanish, French, etc.)

Cannot reliably detect with <5 words
→ Default to English?
→ Try multiple languages?
→ Ask user?

Mixed Language Documents:

Document:
"Introduction [English]
Chapitre 1: Installation [French]
Chapter 2: Configuration [English]
Capítulo 3: Troubleshooting [Spanish]"

What is the document's language?
→ Multi-language
→ Predominant language: English (50%)
→ But important content in others

How to embed?
→ Per-section with language tags?
→ As single multilingual embedding?
→ Multiple embeddings per doc?

Cross-Lingual Search Strategies

Retrieval across language boundaries:

Approach 1: Separate Indexes Per Language:

English index: English docs
Spanish index: Spanish docs
French index: French docs

Query in Spanish:
→ Search Spanish index only
→ Get Spanish results
→ Fast, simple

Limitations:
→ Cannot find related English docs
→ User might benefit from English docs too
→ Knowledge siloed by language

Approach 2: Unified Multilingual Index:

Single index with multilingual embeddings:
→ All docs regardless of language
→ Cross-lingual retrieval possible

Query in Spanish:
→ Retrieve Spanish docs (highest similarity)
→ Also retrieve English docs (lower similarity, but relevant)

User can see both:
→ Preferred language first
→ Other languages as fallback

Hybrid Approach:

Metadata filtering + multilingual search:
1. Detect query language
2. Boost docs in same language (2x multiplier)
3. But still include other languages
4. Present results with language tags

Best of both:
→ Preferred language prioritized
→ Other languages accessible
→ User choice

---

How to Solve

Use multilingual embedding models (mBERT, XLM-RoBERTa) + normalize Unicode encoding (NFC) + implement language detection + boost same-language results but allow cross-lingual retrieval + store language metadata + consider per-section embeddings for mixed-language docs. See Multilingual Search.