Chunking & Processing
Overview
Chunking is the process of breaking down large documents into smaller, semantically meaningful pieces that can be embedded and retrieved effectively. Poor chunking is one of the most common yet overlooked causes of RAG failures—it directly impacts retrieval quality, context coherence, and answer accuracy.
Why Chunking Matters
Effective chunking ensures:
Semantic coherence - Each chunk contains complete, meaningful information
Optimal retrieval - Relevant information can be found and returned
Context preservation - Important relationships and structure are maintained
Efficient embeddings - Chunks fit within token limits while maximizing information density
Poor chunking leads to:
Fragmented context - Critical information split across chunks
Lost structure - Formatting, tables, and code blocks broken apart
Retrieval failures - Relevant information exists but can't be found
Incomplete answers - Agents receive partial context and fill gaps with hallucinations
Common Chunking Challenges
Size Issues
Chunks too small - Context loss, fragmented information, excessive chunks
Chunks too large - Poor retrieval granularity, exceeded token limits
Inconsistent sizing - Variable quality across your knowledge base
Structure Preservation
Code blocks split incorrectly - Broken syntax, incomplete examples
Tables breaking across chunks - Lost relationships between rows/columns
Nested lists fragmented - Hierarchy and context destroyed
Footnotes separated - References disconnected from content
Format Handling
Markdown formatting lost - Headers, emphasis, and structure stripped
Mathematical notation corrupted - LaTeX and formulas broken
Multi-column layouts - Spatial relationships lost in conversion
HTML to text problems - Semantic meaning lost in conversion
Solutions in This Section
Browse these guides to optimize your chunking strategy:
Chunking Strategies
Different content types require different chunking approaches:
By Content Type
Documentation
Semantic splitting by sections
500-1000 tokens
Code
Function/class boundaries
200-800 tokens
Conversations
Message groupings
300-600 tokens
Technical papers
Paragraph-aware splitting
600-1200 tokens
FAQs
Question-answer pairs
100-400 tokens
By Goal
Maximum recall - Smaller chunks (200-400 tokens) with more overlap
Maximum context - Larger chunks (800-1200 tokens) with less fragmentation
Balanced approach - Medium chunks (400-800 tokens) with 10-20% overlap
Best Practices
Respect document structure - Use headers, sections, and natural boundaries
Preserve semantic units - Don't split sentences, code blocks, or tables
Add metadata - Include source, section, and context in chunk metadata
Test retrieval quality - Measure how well chunking supports finding answers
Use overlap strategically - Help bridge context between chunks (10-20%)
Document-type specific strategies - Code needs different handling than prose
Monitor and iterate - Track chunk-level performance and adjust
Impact on RAG Performance
Chunking quality directly affects:
Retrieval Precision
Relevant chunks returned
Irrelevant fragments retrieved
Retrieval Recall
Complete information found
Key details missed
Context Quality
Coherent, complete context
Fragmented, incomplete context
Answer Accuracy
Correct, well-supported answers
Hallucinations to fill gaps
Citation Quality
Clean source attribution
Unclear or broken references
Embedding Quality
Meaningful semantic vectors
Noisy, incoherent embeddings
Quick Diagnostics
Signs your chunking needs work:
✗ Agents give partial answers that feel "cut off"
✗ Retrieved chunks don't contain complete thoughts
✗ Code examples are broken or incomplete
✗ Tables appear without headers or context
✗ References to "see above" or "as mentioned" without the context
✗ High retrieval count but low answer quality
Signs your chunking is working well:
✓ Retrieved chunks are self-contained and meaningful
✓ Agents provide complete, coherent answers
✓ Citations point to relevant, complete content
✓ Code examples are syntactically complete
✓ Tables preserve their structure and relationships
Advanced Considerations
Semantic Chunking
Move beyond fixed token limits to:
Split at semantic boundaries (topic shifts, section changes)
Use embedding similarity to detect natural breakpoints
Preserve logical units (arguments, explanations, examples)
Hierarchical Chunking
Create multi-level chunk hierarchies:
Document level - Overview and metadata
Section level - Major topics and themes
Paragraph level - Specific details and facts
This enables coarse-to-fine retrieval strategies.
Dynamic Chunking
Adjust chunk size based on:
Document type and structure
Content density and complexity
Retrieval performance metrics
User query patterns
Remember: Perfect chunking is content-specific. What works for technical docs may not work for conversational data. Test, measure, and iterate based on your specific use case.
Last updated