API Rate Limit Exhaustion

The Problem

Your data source integration hits rate limits, causing slow syncs, partial data ingestion, or complete sync failures.

Symptoms

• ❌ "429 Too Many Requests" errors

• ❌ Sync takes 10x longer than expected

• ❌ Only syncs 500 docs before stopping

• ❌ Works fine for small datasets, fails for large

• ❌ Other integrations stop working during Twig sync

Real-World Example

Zendesk API rate limit: 700 requests/minute
Your help center: 2,000 articles

Twig sync attempts:
→ Enumerate all categories: 5 requests
→ List articles in each category: 50 requests
→ Fetch full content for each article: 2,000 requests
→ Total: 2,055 requests

At 700/min max: 3 minute minimum

But:
→ Twig makes 50 concurrent requests/sec (3,000/min)
→ After 700 requests: 429 rate limit
→ Twig backs off 60 seconds
→ Resumes, hits limit again
→ Sync takes 45 minutes instead of 3

---

Deep Technical Analysis

Rate Limit Types and Granularity

APIs implement multiple layers of rate limiting:

Per-User Limits:

Limit: 100 requests/minute per user token

If Twig uses API token for [email protected]:
→ All Twig requests count against [email protected]'s quota
→ If user also uses Zendesk app: shares same quota
→ Twig sync can block user's manual API access

Per-App Limits:

Limit: 10,000 requests/hour per OAuth app

If Twig has OAuth app ID: app_123:
→ All requests from Twig app_123 (all customers) share quota
→ 100 customers syncing simultaneously
→ Each uses 100 req/hr → 10,000 total
→ Quota exhausted, all customers affected

Per-Endpoint Limits:

Slack example:
→ conversations.list: 20 req/min (Tier 2)
→ conversations.history: 50 req/min (Tier 3)
→ users.info: 100 req/min (Tier 4)

Twig must:
→ Track quota per endpoint separately
→ Can't assume global rate limit
→ Complex quota bookkeeping

Burst vs Sustained:

Many APIs allow bursts:

Token bucket algorithm:
→ Bucket capacity: 100 tokens
→ Refill rate: 10 tokens/second
→ Each request: consumes 1 token

Burst behavior:
→ 0s: 100 tokens available
→ Make 100 requests instantly: ✓ allowed
→ 1s: 10 tokens refilled
→ Make 10 more: ✓ allowed
→ Attempt 11th: ✗ rate limited

Twig must understand:
→ Initial burst allowance
→ Sustained rate after burst
→ Bucket refill rate

Rate Limit Discovery and Headers

APIs communicate limits via HTTP headers:

Standard Headers (RFCs):

HTTP/1.1 200 OK
X-RateLimit-Limit: 1000
X-RateLimit-Remaining: 742
X-RateLimit-Reset: 1642204800

But variations exist:

GitHub:
→ X-RateLimit-*

Stripe:
→ Stripe-RateLimit-*

Twitter:
→ x-rate-limit-* (lowercase)

Google:
→ No headers (must infer from 429 errors)

Inconsistency:
→ Twig must handle all formats
→ Or: No headers available, must estimate

The Reset Timestamp Ambiguity:

X-RateLimit-Reset: 1642204800

Is this:
→ Unix timestamp (seconds since epoch)?
→ Milliseconds since epoch?
→ Seconds until reset (relative)?
→ ISO 8601 datetime string?

Spec says: Unix timestamp
Reality: Providers differ

Wrong interpretation:
→ Wait 1642204800 seconds (52 years!)
→ Or wait 1 second when should wait 3600

Distributed Rate Limiting

Multiple Twig servers must share quota:

Naive Approach (broken):

3 Twig servers:
→ Server A: tracks "450 requests made"
→ Server B: tracks "380 requests made"
→ Server C: tracks "410 requests made"

Total: 1,240 requests
But actual limit: 1,000 requests/min

Each server thinks it's under limit
All servers make requests
Actual total: Exceeds quota → 429 errors

Centralized Rate Limiter:

Redis-based solution:
→ All servers increment shared counter in Redis
→ INCR api:zendesk:requests:minute_12345
→ SET TTL 60 seconds
→ If counter > 1000: deny request

Pros:
+ Accurate shared state
+ Prevents over-limit requests

Cons:
- Redis becomes bottleneck (1ms latency per request)
- Redis failure blocks all API calls
- Network overhead

The Race Condition:

Two servers simultaneously:

Server A:
1. GET counter → 999
2. Check: 999 < 1000 → OK
3. Make API call
4. INCR counter → 1000

Server B (concurrent):
1. GET counter → 999 (before Server A's INCR)
2. Check: 999 < 1000 → OK
3. Make API call
4. INCR counter → 1001

Both requests approved, but limit is 1000
Result: Quota exceeded by 1

Solution: Atomic operation (INCR then check)
But: Check happens after INCR, too late to prevent

Retry and Backoff Strategies

Handling 429 responses requires exponential backoff:

Naive Retry (bad):

Request fails with 429:
→ Wait 1 second
→ Retry
→ 429 again
→ Wait 1 second
→ Retry...
→ Hammering API with retries
→ Makes rate limiting worse

Exponential Backoff (better):

Attempt 1: Immediate request → 429
→ Wait 1 second
Attempt 2: Retry → 429
→ Wait 2 seconds
Attempt 3: Retry → 429
→ Wait 4 seconds
Attempt 4: Retry → 429
→ Wait 8 seconds
Attempt 5: Retry → Success

Respects provider's recovery time

Jitter to Prevent Thundering Herd:

Without jitter:
→ 100 requests hit limit at same time
→ All wait exactly 8 seconds
→ All retry at exact same moment
→ Thundering herd, 429 again

With jitter:
wait_time = 8 + random(0, 2)
→ Request A: waits 8.3s
→ Request B: waits 9.1s
→ Request C: waits 8.7s
→ Spread out retries, smoother traffic

The Give-Up Point:

How long to retry?

Option 1: Fixed attempts (e.g., 5 retries)
→ Total time: 1 + 2 + 4 + 8 + 16 = 31 seconds
→ If still failing: give up

Option 2: Fixed duration (e.g., 5 minutes)
→ Keep retrying until 5 min elapsed
→ Then fail

Option 3: Retry-After header
→ 429 response includes: Retry-After: 120
→ Wait exactly 120 seconds
→ One retry, then give up if still failing

Best: Respect Retry-After, fallback to exponential backoff

Request Batching and Pagination

Reducing API calls through efficient patterns:

The Pagination Efficiency Problem:

List 5,000 articles:

Approach A (inefficient):
→ Page size: 10
→ 500 pages × 1 request each = 500 API calls

Approach B (efficient):
→ Page size: 100
→ 50 pages × 1 request each = 50 API calls

10x fewer requests, same data

But maximum page size limits:

Most APIs:
→ Max page size: 100 (Google Drive)
→ Max page size: 50 (Zendesk)
→ No way to request more per call

Twig must:
→ Use largest allowed page size
→ Minimize paginated requests

Batching Where Supported:

Some APIs allow batch requests:

GitHub GraphQL:
query {
  repo1: repository(owner: "x", name: "y") { ... }
  repo2: repository(owner: "a", name: "b") { ... }
  repo3: repository(owner: "c", name: "d") { ... }
}

Single request fetches 3 repositories
→ 3x efficiency vs REST

But:
- Not all APIs support batching
- GraphQL has query complexity limits
- Still counts as multiple units toward quota

Quota Pooling and Reservation

Advanced rate limit management:

The Multi-Data-Source Problem:

Customer has:
→ Data Source A: needs 500 req/min
→ Data Source B: needs 300 req/min
→ Data Source C: needs 400 req/min
Total: 1,200 req/min

But API quota: 1,000 req/min

All three sources can't sync simultaneously
Must coordinate:
→ Source A: 500 req/min
→ Source B: 300 req/min
→ Source C: 200 req/min (throttled)
→ After A finishes: C gets remaining quota

Priority-Based Scheduling:

Assign priorities:
1. Real-time updates (webhook-triggered): High priority
2. Incremental syncs (hourly): Medium priority
3. Full syncs (daily): Low priority

Quota allocation:
→ Reserve 30% for high priority
→ 50% for medium
→ 20% for low

If high priority needs more:
→ Borrow from medium/low
→ Ensure critical updates never blocked

The Cold Start Problem:

New customer connects Zendesk:
→ 10,000 articles to sync (initial full sync)
→ At 700 req/min: 15 minute sync time

But:
→ Existing customers' incremental syncs running
→ Using 600/700 req/min quota
→ New customer gets 100 req/min remaining
→ Sync takes 100 minutes instead of 15

Solution:
→ Queue new customer's full sync for off-peak hours
→ Or: Throttle existing customers temporarily
→ Tradeoff: New customer waits longer

Cost vs Speed Tradeoffs

Respecting rate limits reduces sync speed:

The Impatience Problem:

User connects data source at 10:00 AM
Expects: Data ready in 5 minutes
Reality: Rate limits mean 45 minute sync

User perception:
→ "This is so slow!"
→ "Is it stuck?"
→ Cancels sync
→ Incomplete knowledge base

Paid Tiers for Higher Limits:

Many APIs offer:
→ Free tier: 100 req/min
→ Pro tier: 1,000 req/min
→ Enterprise tier: 10,000 req/min

Twig options:
1. Absorb cost: Upgrade all customers to Pro
   → Expensive at scale

2. Pass through cost: Customer pays for their API tier
   → Better user experience if they upgrade
   → But complexity: billing, provisioning

3. Optimize within free tier
   → Slower but free
   → Current approach for most

---

How to Solve

Implement exponential backoff with jitter + respect Retry-After headers + use Redis for distributed rate limiting + batch requests where possible + use largest page sizes + reserve quota for high-priority updates. See Rate Limit Handling.