Embedding Cost Optimization

The Problem

Embedding API costs scale with document volume and updates—large knowledge bases or frequent changes result in expensive monthly bills.

Symptoms

• ❌ $500+/month embedding costs

• ❌ Re-embedding on every doc update

• ❌ Charges for unchanged content

• ❌ Costs grow linearly with content

• ❌ No cost visibility or control

Real-World Example

Knowledge base: 10,000 documents
Average size: 2,000 tokens per doc
Total: 20 million tokens

Monthly updates: 30% of docs change
→ 3,000 docs × 2,000 tokens = 6 million tokens/month

OpenAI embedding cost: $0.0001 per 1K tokens

Initial embedding: 20M tokens = $2
Monthly re-embedding: 6M tokens = $0.60

Yearly cost: $2 + (12 × $0.60) = $9.20

Seems cheap, but:
→ 1M documents = $920/year
→ 100 customers = $92K/year
→ Significant at scale

---

Deep Technical Analysis

Token Counting and Pricing

Understanding cost calculation:

Tokenization Overhead:

Document text: 1,000 words
≈ 1,300 tokens (typical English ratio: 1.3:1)

Why more tokens than words?
→ Subword tokenization
→ Punctuation as separate tokens
→ "don't" → ["don", "'", "t"]

Cost based on tokens, not words
→ Must count tokens accurately
→ Use tiktoken library (OpenAI)

Batch Processing Discounts:

Some providers offer:
→ Volume discounts (>10M tokens/month)
→ Batch API endpoints (cheaper but slower)

OpenAI Batch API:
→ 50% discount
→ But: 24-hour SLA
→ Not suitable for real-time

Trade-off:
→ Save money vs real-time processing

Deduplication and Caching

Avoid re-embedding identical content:

Content Hashing:

Before embedding:
1. Compute hash of document text
   → SHA-256(content) = "abc123..."
2. Check cache: Is "abc123" already embedded?
3. If yes: Reuse existing embedding
4. If no: Call API, embed, cache result

Savings:
→ Document updated: Metadata only (title, author)
→ Content unchanged
→ Reuse cached embedding
→ $0 cost

Chunk-Level Caching:

Document: 10 chunks

User updates paragraph 3 (chunk 3)
→ Chunks 1,2,4,5,6,7,8,9,10 unchanged

Smart re-embedding:
→ Reuse embeddings for 9 unchanged chunks
→ Only embed chunk 3
→ 90% cost savings

Challenge:
→ Chunk boundaries may shift
→ Paragraph 3 edit affects chunk 4
→ Must detect boundary changes

The Boundary Shift Problem:

Original chunking (512 tokens each):
→ Chunk 1: Tokens 1-512
→ Chunk 2: Tokens 513-1024
→ Chunk 3: Tokens 1025-1536

User adds 100 tokens to chunk 1:
→ Chunk 1: Tokens 1-612 (grew)
→ Chunk 2: Tokens 613-1124 (shifted!)
→ Chunk 3: Tokens 1125-1636 (shifted!)

All chunk boundaries changed
→ Must re-embed all chunks
→ Cannot reuse cache

Solution:
→ Use semantic chunking (section-based)
→ Boundaries don't shift as easily

Incremental Updates

Only process changed content:

Document-Level Tracking:

Store metadata:
→ doc_id: "guide_123"
→ content_hash: "abc123"
→ last_embedded: "2024-01-15"

On update:
1. Fetch doc_id from source
2. Compute new hash: "def456"
3. Compare: "abc123" vs "def456"
4. If different: Re-embed
5. If same: Skip (metadata-only change)

Avoids unnecessary API calls

The Last-Modified Trap:

Source system provides: last_modified timestamp

Naive check:
→ If last_modified > last_embedded: Re-embed

Problem:
→ User opens doc, saves without changes
→ last_modified updates
→ Triggers re-embedding
→ Wasted cost

Better:
→ Content hash comparison
→ Only re-embed if hash changed

Token Optimization Techniques

Reduce token count without losing meaning:

Whitespace Normalization:

Original:
"The    API    has     multiple spaces"
→ 8 tokens (spaces tokenized)

Normalized:
"The API has multiple spaces"
→ 5 tokens

Savings: 37% (for this extreme case)

Boilerplate Removal:

HTML extraction includes:
"Copyright © 2024 Company Inc. All rights reserved."

Appears in every document footer:
→ 10 tokens × 1,000 docs = 10K tokens
→ No semantic value
→ Pure cost

Remove before embedding:
→ $0.001 savings (small per-doc)
→ $1 savings at 1M docs

Code Block Optimization:

Code example:
```python
def authenticate(username, password):
    # Validate credentials
    if not username or not password:
        raise ValueError("Missing credentials")
    
    # Query database
    user = db.query(User).filter_by(username=username).first()
    
    # Verify password
    if verify_password(password, user.password_hash):
        return generate_token(user)
    raise AuthenticationError()

Tokens: 120

Summarized for embedding: "Python function: authenticate(username, password). Validates credentials, queries database, verifies password, returns token or raises error."

Tokens: 25

Savings: 79%

Trade-off: Lose exact code, keep semantic meaning

### Model Selection

Cheaper models for appropriate use cases:

**Cost Comparison:**

OpenAI text-embedding-ada-002: → $0.0001 per 1K tokens → 1536 dimensions → High quality

Cohere embed-english-light-v3.0: → $0.00002 per 1K tokens (5x cheaper!) → 384 dimensions → Good quality

Sentence-BERT (self-hosted): → Free (after compute costs) → 768 dimensions → Decent quality

Decision matrix: → Critical docs: Use best model → FAQ/simple content: Use cheaper model → High-volume/low-value: Self-host

**Hybrid Model Strategy:**

Tier content by importance: → Tier 1 (20%): Product docs, critical guides → Use OpenAI ada-002 ($$$) → Tier 2 (50%): General documentation → Use Cohere light ($$) → Tier 3 (30%): FAQs, old content → Use self-hosted Sentence-BERT ($)

Weighted cost optimization

### Self-Hosting Considerations

Running own embedding models:

**Cost Analysis:**

Self-hosted setup: → GPU instance: $500/month (AWS p3.2xlarge) → Can embed ~50M tokens/month → Effective cost: $0.00001 per 1K tokens

vs.

OpenAI API: → $0.0001 per 1K tokens → 50M tokens = $5,000/month

Break-even: 5K tokens/month → Self-host only if high volume

**Hidden Costs:**

Self-hosting requires: → Model ops expertise → Infrastructure maintenance → Monitoring and alerts → Model updates/versioning → Scaling during spikes

Labor: $10K/month (engineer time) → Only worth it at very high scale → >500M tokens/month

### Rate Limiting and Throttling

Manage API usage:

**Burst Control:**

User uploads 1,000 docs at once: → 1,000 embed API calls immediately → Exceeds rate limit (3,500/min) → 429 errors

Better approach: → Queue documents → Process at 50 docs/min → Takes 20 minutes → Smooth cost distribution → No rate limit errors

**Budget Caps:**

Set monthly budget: $100

Track spend: → Real-time counter → When approaching $95: Slow down → At $100: Pause embedding → Alert admin

Prevents runaway costs → Protects from accidental overspending

---

## How to Solve

**Implement content hashing for deduplication + cache embeddings with cache-aside pattern + use semantic chunking to minimize boundary shifts + normalize whitespace and remove boilerplate + consider cheaper models (Cohere light) for non-critical content + set budget caps and rate limits.** See [Embedding Cost Management](../vectors/embedding-costs.md).