Markdown Formatting Lost

The Problem

Markdown formatting (bold, italic, links, code) is stripped during chunking, losing semantic emphasis and making retrieved content less useful.

Symptoms

• ❌ code blocks render as plain text

• ❌ bold emphasis lost in answers

• ❌ Links converted to bare URLs or text

• ❌ > Blockquotes not distinguished from body text

• ❌ Headers (##) indistinguishable from paragraphs

Real-World Example

Original markdown:
## Authentication Methods

We support **three authentication methods**:

1. **API Key**: Include `X-API-Key` header
2. **OAuth 2.0**: Use `/oauth/token` endpoint  
3. **JWT**: Bearer token in `Authorization` header

See [Authentication Guide](/docs/auth) for details.

After naive text extraction:
Authentication Methods We support three authentication methods: API Key: Include X-API-Key header OAuth 2.0: Use /oauth/token endpoint JWT: Bearer token in Authorization header See Authentication Guide for details.

Lost:
- Header hierarchy (##)
- Bold emphasis (**important**)
- Inline code (`code`)
- List structure (1, 2, 3)
- Link destination ([text](url))

---

Deep Technical Analysis

Semantic Information in Formatting

Markdown formatting carries meaning:

Format Types and Semantics:

**bold** / __bold__:
→ Emphasis, importance, key terms
→ "The **API rate limit** is 1000/hour"
→ "API rate limit" is the critical information

*italic* / _italic_:
→ Subtle emphasis, definitions, foreign terms
→ "The *Bearer* token goes in the header"

`code`:
→ Technical terms, literals, exact syntax
→ "Set `DEBUG=true` in config"
→ Indicates must use exact string

[link text](url):
→ Reference to external resource
→ Anchor text vs destination
→ "See [API docs](https://api.example.com)"

The Loss of Signal:

Original: "Use the **POST** method, not GET"
Extracted: "Use the POST method, not GET"

Without bold:
→ LLM can't tell POST is emphasized
→ Might treat "POST" and "GET" equally
→ User wanted explicit guidance: "Use POST (important!)"

Similarly:
"Configure `max_connections=100` in settings"
→ Becomes: "Configure max_connections=100 in settings"
→ Lost signal that it's a literal config key

Conversion Strategies

Different approaches to handling markdown:

Strategy 1: Strip All Formatting (current)

Markdown → Plain text

Input: "The **API key** goes in the `X-API-Key` header"
Output: "The API key goes in the X-API-Key header"

Pros:
+ Simple, fast
+ Smaller text (fewer tokens)

Cons:
- Loses semantic emphasis
- Code not distinguished from prose
- Links lose destination

Strategy 2: Convert to Natural Language:

Markdown → Descriptive text

Input: "Use the **POST** method"
Output: "Use the POST method (emphasized)"

Input: "Set `DEBUG=true`"
Output: "Set the configuration value DEBUG to true"

Input: "[API docs](https://api.example.com)"
Output: "See API docs at https://api.example.com"

Pros:
+ Preserves semantic meaning
+ More verbose but clearer

Cons:
- Significantly more tokens
- Conversions may be awkward
- Not standard format

Strategy 3: Keep Markdown Syntax:

Markdown → Markdown (as-is)

Input: "Use the **POST** method"
Output: "Use the **POST** method"

Pros:
+ Preserves all formatting
+ LLMs trained on markdown (GPT-4, etc.)
+ Reversible (can render to HTML later)

Cons:
- Extra characters (**, `, etc.)
- May confuse simpler models
- Embeddings see formatting chars as content

Strategy 4: Convert to HTML:

Markdown → HTML

Input: "Use the **POST** method"
Output: "Use the <strong>POST</strong> method"

Input: "Set `DEBUG=true`"
Output: "Set <code>DEBUG=true</code>"

Pros:
+ Standard format
+ Preserves structure
+ Can render in UI

Cons:
- Verbose (<strong>, </strong>, etc.)
- More tokens than markdown
- HTML tags in embeddings

Inline Code vs Code Blocks

Different code representations need different handling:

Inline Code:

Use the `Authorization` header with value `Bearer {token}`.

Preservation Importance:

Without backticks:
"Use the Authorization header with value Bearer {token}."

Problems:
→ "Bearer {token}" might be interpreted as prose
→ LLM might rephrase as "Bearer token"
→ User needs exact literal: "Bearer {token}"

With backticks preserved:
"Use the `Authorization` header with value `Bearer {token}`."

LLM recognizes:
→ These are literal strings
→ Don't paraphrase
→ Use exactly as written

Code Blocks:

```python
def authenticate(api_key):
    headers = {"Authorization": f"Bearer {api_key}"}
    return requests.post(url, headers=headers)
```

Extraction Challenge:

Should we:
1. Keep triple-backticks and language identifier?
   → "```python\ndef authenticate..."
   → Preserves language context
   → But: Extra tokens

2. Strip fences, keep code?
   → "def authenticate(api_key):..."
   → Cleaner, fewer tokens
   → But: Lost language info

3. Add language metadata separately?
   → Text: "def authenticate..."
   → Metadata: {language: "python"}
   → Clean text, structured metadata
   → Requires metadata support

Link Handling and References

Links have both anchor text and destination:

Link Structure:

[GitHub Repository](https://github.com/company/repo)

Extraction Options:

Option 1: Keep anchor text only
→ "GitHub Repository"
→ Readable, concise
→ But: URL lost, can't navigate

Option 2: Keep URL only
→ "https://github.com/company/repo"
→ URL preserved
→ But: Lost descriptive text

Option 3: Expand to "text (url)"
→ "GitHub Repository (https://github.com/company/repo)"
→ Both preserved
→ But: More verbose

Option 4: Keep markdown syntax
→ "[GitHub Repository](https://github.com/company/repo)"
→ Full information
→ But: Extra characters

Reference Link Problem:

See the [API Guide][1] for details.

[1]: https://docs.example.com/api

Reference Resolution:

Reference defined elsewhere in document:
→ May be in different chunk
→ Must resolve before chunking
→ Or: Store unresolved references

Without resolution:
Chunk 1: "See the [API Guide][1] for details."
Chunk 2: "[1]: https://docs.example.com/api"

LLM sees Chunk 1:
→ "[1]" reference not resolved
→ Incomplete information

List Structure Preservation

Lists have hierarchical structure:

Ordered Lists:

1. First step
2. Second step
   1. Sub-step A
   2. Sub-step B
3. Third step

Flattening Problem:

Naive extraction:
"First step Second step Sub-step A Sub-step B Third step"

Lost:
→ Numbering (sequence matters)
→ Hierarchy (2.1, 2.2 are children of 2)
→ Step boundaries

Better extraction:
"Step 1: First step. Step 2: Second step (Sub-step A, Sub-step B). Step 3: Third step."

Preserves:
→ Sequence
→ Hierarchy (parenthetical sub-steps)
→ Boundaries

Unordered Lists:

Features:
- Fast performance
- Easy to use
- Highly scalable

Structure Loss:

Flattened:
"Features: Fast performance Easy to use Highly scalable"

Better:
"Features: (1) Fast performance, (2) Easy to use, (3) Highly scalable"

Or:
"Features include: Fast performance. Easy to use. Highly scalable."

Blockquotes and Callouts

Special blocks carry semantic meaning:

Blockquote:

> **Important**: Always validate user input before processing.

Semantic Significance:

Without formatting:
"Important: Always validate user input before processing."

Lost: Blockquote indicates this is advisory/warning
→ Not just normal prose
→ Carries weight, should be emphasized

With preservation:
"Note (blockquote): Important: Always validate user input before processing."

Or keep markdown:
"> **Important**: Always validate user input before processing."

Callout Boxes (GitHub-flavored):

> [!WARNING]
> This action cannot be undone!

> [!NOTE]
> This is useful information.

Semantic Type Loss:

Extracted as plain text:
"This action cannot be undone! This is useful information."

Lost distinction:
→ WARNING (critical) vs NOTE (informational)
→ Both treated equally

Better:
"Warning: This action cannot be undone."
"Note: This is useful information."

Header Hierarchy

Headers define document structure:

Markdown Headers:

# Main Title
## Section
### Subsection
#### Sub-subsection

Structural Information:

Flattened:
"Main Title Section Subsection Sub-subsection"

All run together, no hierarchy

Better approach:
"Document: Main Title > Section > Subsection > Sub-subsection"

Or preserve markdown:
"# Main Title\n## Section\n### Subsection"

Headers tell LLM:
→ Document structure
→ Topic hierarchy
→ Importance (# > ## > ###)

Embedding Model Considerations

How models handle formatted text:

Training Data Exposure:

LLMs (GPT-4, Claude) trained on:
→ Lots of markdown (GitHub, docs)
→ Understand **bold**, `code`, etc.
→ Can interpret formatting natively

Embedding models (ada-002) trained on:
→ Mixed text (markdown + plain)
→ Uncertain how well they handle formatting

Question:
→ Does "Use the **POST** method" embed differently than "Use the POST method"?
→ Does ** add noise or helpful signal?
→ No clear answer, model-dependent

Token Efficiency:

Plain text: "Set DEBUG to true" (4 tokens)
Markdown: "Set `DEBUG` to `true`" (8 tokens with backticks)

2x tokens for same semantic content
→ Higher embedding cost
→ Fills context window faster

Trade-off:
→ Semantic precision vs token efficiency

---

How to Solve

Keep markdown syntax intact for LLM consumption + convert special blocks (warnings, notes) to natural language labels + preserve inline code backticks + expand links to "text (URL)" format + maintain list numbering with explicit labels. See Markdown Handling.