Token Limit Exceeded

The Problem

Generated responses hit model token limits mid-answer, cutting off responses or preventing generation entirely.

Symptoms

• ❌ Responses end abruptly mid-sentence

• ❌ "Maximum tokens reached" errors

• ❌ Incomplete lists or code examples

• ❌ Must request "continue" from user

• ❌ Cannot generate long-form content

Real-World Example

User asks: "List all API endpoints"
AI starts response:
"Here are the API endpoints:
1. POST /auth/login - User authentication
2. GET /users - Retrieve user list
3. POST /users - Create new user
4. GET /users/{id} - Get user details
5. PUT /users/{id} - Update user
..." 

[Token limit reached at 1000 tokens]

Response cuts off at endpoint #15 of 50 total
User sees incomplete list

---

Deep Technical Analysis

Output Token Limits

Separate from input context limits:

Max Tokens Parameter:

API calls specify max_tokens:
→ GPT-4: max_tokens=4096
→ Controls response length
→ Prevents runaway generation

Trade-offs:
→ Too low: Truncated responses
→ Too high: Longer latency, higher cost
→ Must balance

Automatic Truncation:

LLM generates token-by-token until:
1. Reaches natural stop (EOS token)
2. Hits max_tokens limit
3. Encounters stop sequence

If hits #2 mid-generation:
→ Cuts off wherever it stopped
→ No graceful ending
→ Incomplete output

Estimating Response Length

Predicting token needs:

Query Type Heuristics:

Factual query: "What is X?"
→ Expected: 50-200 tokens
→ Set max_tokens: 300 (buffer)

List query: "List all..."
→ Unknown length
→ Set high limit or paginate

Explanatory: "How does X work?"
→ Expected: 300-800 tokens
→ Set max_tokens: 1000

Dynamic Allocation:

Analyze query:
→ Count items in retrieved context
→ "50 API endpoints found"
→ Estimate: 50 × 30 tokens/item = 1500 tokens
→ Set max_tokens: 2000

Adaptive based on content

Pagination Strategies

Breaking responses into chunks:

Explicit Pagination:

System prompt: "If response exceeds 800 tokens, end with
[Continued in next message] and stop."

User experience:
→ AI sends first part
→ User clicks "Continue"
→ AI resumes with context

Preserves continuity across messages

Automatic Chunking:

Backend splits long responses:
1. Generate full response (internally)
2. Split at natural boundaries (paragraphs)
3. Send as multiple messages
4. Stream to user sequentially

Transparent to user

Summarization vs Detail

Adjusting verbosity:

Conciseness Prompting:

Add to system prompt:
"Be concise. Provide direct answers without unnecessary
elaboration."

Reduces token usage:
→ Same information
→ Fewer words
→ Fits in token budget

Detail Level Control:

User specifies preference:
→ "Give brief overview" (200 tokens)
→ "Explain in detail" (1000 tokens)

Adjust max_tokens accordingly

Token Accounting

Tracking usage:

Input + Output Budget:

Total model capacity: 8K tokens

Input (6K tokens):
→ System prompt: 300
→ Context: 5,500
→ Query: 200

Remaining: 2,000 tokens
→ Maximum possible response length
→ Set max_tokens ≤ 2,000

Conversation History:

Multi-turn chat accumulates:
→ Turn 1: 500 tokens (in + out)
→ Turn 2: 600 tokens
→ Turn 3: 700 tokens
→ Total: 1,800 tokens in history

Context window filling up:
→ Less space for future responses
→ Must prune old turns

Response Compression

Fitting more in less space:

Structured Formats:

Instead of prose:
"The API rate limit is 1000 requests per hour. If you 
exceed this limit, you will receive a 429 error..."

Use structured:
{
  "rate_limit": "1000/hour",
  "error_code": 429,
  "retry_after": "60 seconds"
}

Same info, fewer tokens

Tables Over Lists:

Verbose list (200 tokens):
"Endpoint 1: POST /auth/login - Used for authentication...
Endpoint 2: GET /users - Retrieves user list..."

Table (120 tokens):
| Method | Path | Description |
|--------|------|-------------|
| POST | /auth/login | Authentication |
| GET | /users | User list |

---

How to Solve

Set max_tokens dynamically based on query type + implement response pagination for long outputs + use conciseness prompts to reduce verbosity + employ structured formats (tables, JSON) over prose + track token usage and adjust context accordingly. See Token Management.