# Temperature Setting Issues

## The Problem

Incorrect temperature settings cause either robotic, repetitive responses or creative but factually incorrect answers, degrading RAG quality.

### Symptoms

* ❌ Identical phrasing for similar queries
* ❌ Overly formal/robotic responses
* ❌ Creative but wrong information added
* ❌ Inconsistent response style
* ❌ Cannot balance accuracy vs natural language

### Real-World Example

```
Temperature 0.0 (deterministic):
Query 1: "How to authenticate?"
Response: "To authenticate, use the API key in the Authorization header."

Query 2: "What's the auth method?"
Response: "To authenticate, use the API key in the Authorization header."

Exact same wording → robotic

Temperature 1.5 (creative):
Query: "API rate limit?"
Response: "The API implements a sophisticated adaptive rate limiting
system that adjusts based on your usage patterns and account tier,
typically allowing between 800-1200 requests per hour..."

Added details not in context → hallucination
```

***

## Deep Technical Analysis

### Temperature Parameter Mechanics

Controls randomness in token selection:

**How It Works:**

```
LLM outputs probability distribution over vocabulary:
→ "authenticate": 0.45 (highest)
→ "verify": 0.25
→ "authorize": 0.15
→ "confirm": 0.10
→ ...

Temperature = 0: Always pick highest (deterministic)
→ "authenticate" every time

Temperature = 1: Sample from distribution
→ "authenticate" 45% of time
→ "verify" 25% of time

Temperature = 2: Flatten distribution
→ More uniform sampling
→ Higher diversity
```

### RAG-Specific Considerations

RAG needs factual accuracy:

**Low Temperature (0.0-0.3) Benefits:**

```
Pros:
+ Consistent responses
+ Factually grounded
+ Reproducible
+ Less hallucination risk

Cons:
- Repetitive phrasing
- Robotic tone
- Lacks conversational flow
- Same intro every time
```

**High Temperature (0.8-1.5) Risks:**

```
Pros:
+ Natural language
+ Varied phrasing
+ Conversational tone

Cons:
- Creative additions (hallucination)
- Paraphrases may drift from facts
- Less controllable
- Inconsistent quality
```

### The Sweet Spot

Balancing accuracy and fluency:

**Temperature 0.3-0.5:**

```
Optimal for RAG:
→ Mostly follows context
→ Some variation in phrasing
→ Natural but grounded
→ Minimal hallucination

Testing shows:
→ 0.3: Very safe, slightly robotic
→ 0.5: Good balance
→ 0.7: More creative, higher risk
```

**Context-Dependent Adjustment:**

```
Factual queries: temperature=0.2
→ "What is the rate limit?"
→ Exact fact needed

Explanatory queries: temperature=0.5
→ "How does authentication work?"
→ More natural explanation acceptable

Creative queries: temperature=0.7
→ "Explain this concept to a beginner"
→ Pedagogical flexibility useful
```

### Top-P (Nucleus) Sampling

Alternative to temperature:

**Top-P Mechanism:**

```
Instead of temperature, use top_p:
→ Sample from smallest set of tokens where cumulative
  probability ≥ p

top_p=0.9:
→ "authenticate" (0.45) + "verify" (0.25) + "authorize" (0.15) 
  + others = 0.90
→ Only consider these tokens
→ Ignore very low probability options

More controlled than high temperature
```

**Combined Settings:**

```
temperature=0.7, top_p=0.9:
→ Soften probabilities (temperature)
→ But cap at top 90% (top_p)
→ Balanced creativity with safety

Recommended for RAG:
→ temperature=0.5
→ top_p=0.95
```

***

## How to Solve

**Set temperature=0.3-0.5 for RAG (factual grounding) + use top\_p=0.9-0.95 for additional control + adjust per query type (lower for factual, higher for explanatory) + test with eval set to find optimal balance + never exceed 0.7 for production RAG.** See [Temperature Tuning](/rag-scenarios-and-solutions/llm/temperature-tuning.md).


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://help.twig.so/rag-scenarios-and-solutions/llm/temperature-tuning.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.