Character Encoding in Chunks

The Problem

Text with special characters, emojis, or non-ASCII content breaks during embedding or retrieval, causing garbled text in AI responses.

Symptoms

• ❌ Foreign language text displays as "???" or boxes

• ❌ Emojis become broken characters

• ❌ Math symbols corrupted

• ❌ Smart quotes become weird chars

• ❌ Retrieval fails on non-ASCII queries

Real-World Example

Source document (UTF-8):
"Price: €500 for 10× improvement 🚀"

After ingestion (wrong encoding):
"Price: €500 for 10× improvement ?"

AI response cites corrupted text:
→ User sees garbage characters
→ Cannot understand pricing
→ Trust degraded

---

Deep Technical Analysis

Encoding Mismatches

UTF-8 vs Latin-1:

Document encoded as UTF-8:
→ "café" = [0xC3, 0xA9, 0x66, 0xC3, 0xA9]

Read as Latin-1 (wrong):
→ Interprets UTF-8 bytes as Latin-1
→ Displays: "café"

Must detect/declare encoding correctly

Windows-1252 vs UTF-8:

Smart quotes (Word docs):
→ " " (curly quotes)
→ Windows-1252 encoding

If treated as UTF-8:
→ Displays as � or ?
→ Common with Office doc imports

Embedding Model Limitations

Model Vocabulary:

Some embedding models:
→ Trained primarily on English ASCII
→ Limited non-ASCII support
→ May handle poorly:
  - Chinese characters
  - Arabic script
  - Cyrillic
  - Emoji

Result: Poor embeddings for non-English

Normalization:

Pre-process before embedding:
→ Convert smart quotes to straight quotes
→ " " → " "
→ – (en-dash) → - (hyphen)

Reduces encoding issues
But: Loses semantic nuance

Detection Strategies

Encoding Detection:

Use chardet library (Python):
→ Detects encoding probabilistically
→ "This looks like UTF-8 with 95% confidence"

Apply detected encoding:
→ Decode file correctly
→ Re-encode as UTF-8 standard

Prevents misinterpretation

Validation:

After ingestion, check:
→ Any � (replacement character)?
→ Excessive non-ASCII ranges?
→ Flag for review

Alerts to encoding problems

PDF Extraction Issues

OCR vs Native Text:

PDFs with scanned images:
→ OCR extracts text
→ OCR errors common:
  - l vs I vs 1 (ambiguous)
  - o vs 0
  - Special chars misread

Native PDFs (better):
→ Embedded text preserved
→ Higher fidelity

Font Encoding:

Custom fonts in PDFs:
→ Character mapping non-standard
→ Extraction gives wrong characters

Example:
→ Displays "А" (Cyrillic A)
→ Extracts "A" (Latin A)
→ Looks same, semantically different

---

How to Solve

Detect encoding with chardet before processing + standardize to UTF-8 for all content + normalize problematic characters (smart quotes, dashes) + use multilingual embedding models for non-English + validate extracted text for replacement characters + prefer native text PDF over OCR when possible + test with multi-language eval set. See Character Encoding.