Temperature Setting Issues

The Problem

Incorrect temperature settings cause either robotic, repetitive responses or creative but factually incorrect answers, degrading RAG quality.

Symptoms

• ❌ Identical phrasing for similar queries

• ❌ Overly formal/robotic responses

• ❌ Creative but wrong information added

• ❌ Inconsistent response style

• ❌ Cannot balance accuracy vs natural language

Real-World Example

Temperature 0.0 (deterministic):
Query 1: "How to authenticate?"
Response: "To authenticate, use the API key in the Authorization header."

Query 2: "What's the auth method?"
Response: "To authenticate, use the API key in the Authorization header."

Exact same wording → robotic

Temperature 1.5 (creative):
Query: "API rate limit?"
Response: "The API implements a sophisticated adaptive rate limiting
system that adjusts based on your usage patterns and account tier,
typically allowing between 800-1200 requests per hour..."

Added details not in context → hallucination

---

Deep Technical Analysis

Temperature Parameter Mechanics

Controls randomness in token selection:

How It Works:

LLM outputs probability distribution over vocabulary:
→ "authenticate": 0.45 (highest)
→ "verify": 0.25
→ "authorize": 0.15
→ "confirm": 0.10
→ ...

Temperature = 0: Always pick highest (deterministic)
→ "authenticate" every time

Temperature = 1: Sample from distribution
→ "authenticate" 45% of time
→ "verify" 25% of time

Temperature = 2: Flatten distribution
→ More uniform sampling
→ Higher diversity

RAG-Specific Considerations

RAG needs factual accuracy:

Low Temperature (0.0-0.3) Benefits:

Pros:
+ Consistent responses
+ Factually grounded
+ Reproducible
+ Less hallucination risk

Cons:
- Repetitive phrasing
- Robotic tone
- Lacks conversational flow
- Same intro every time

High Temperature (0.8-1.5) Risks:

Pros:
+ Natural language
+ Varied phrasing
+ Conversational tone

Cons:
- Creative additions (hallucination)
- Paraphrases may drift from facts
- Less controllable
- Inconsistent quality

The Sweet Spot

Balancing accuracy and fluency:

Temperature 0.3-0.5:

Optimal for RAG:
→ Mostly follows context
→ Some variation in phrasing
→ Natural but grounded
→ Minimal hallucination

Testing shows:
→ 0.3: Very safe, slightly robotic
→ 0.5: Good balance
→ 0.7: More creative, higher risk

Context-Dependent Adjustment:

Factual queries: temperature=0.2
→ "What is the rate limit?"
→ Exact fact needed

Explanatory queries: temperature=0.5
→ "How does authentication work?"
→ More natural explanation acceptable

Creative queries: temperature=0.7
→ "Explain this concept to a beginner"
→ Pedagogical flexibility useful

Top-P (Nucleus) Sampling

Alternative to temperature:

Top-P Mechanism:

Instead of temperature, use top_p:
→ Sample from smallest set of tokens where cumulative
  probability ≥ p

top_p=0.9:
→ "authenticate" (0.45) + "verify" (0.25) + "authorize" (0.15) 
  + others = 0.90
→ Only consider these tokens
→ Ignore very low probability options

More controlled than high temperature

Combined Settings:

temperature=0.7, top_p=0.9:
→ Soften probabilities (temperature)
→ But cap at top 90% (top_p)
→ Balanced creativity with safety

Recommended for RAG:
→ temperature=0.5
→ top_p=0.95

---

How to Solve

Set temperature=0.3-0.5 for RAG (factual grounding) + use top_p=0.9-0.95 for additional control + adjust per query type (lower for factual, higher for explanatory) + test with eval set to find optimal balance + never exceed 0.7 for production RAG. See Temperature Tuning.