Hallucination in Responses

The Problem

LLMs generate confident but factually incorrect information not present in retrieved context, leading to wrong answers despite having correct source material.

Symptoms

• ❌ AI invents facts not in retrieved documents

• ❌ Confident responses with fabricated details

• ❌ Mixing real and made-up information

• ❌ Cannot cite source for invented claims

• ❌ Plausible-sounding but wrong answers

Real-World Example

Retrieved context: "API rate limit is 1000 requests per hour"

User query: "What happens if I exceed the rate limit?"

AI response: "If you exceed the rate limit of 1000 requests per hour, 
your account will be temporarily suspended for 15 minutes and you'll 
receive a 429 error. After three violations, your API key will be 
permanently revoked."

Problem: Context only mentions the limit
→ "15 minutes suspension" - INVENTED
→ "three violations" policy - INVENTED  
→ "permanent revocation" - INVENTED

Only "1000 requests/hour" and "429 error" might be accurate

---

Deep Technical Analysis

Retrieval-Generation Gap

LLM operates beyond retrieved context:

Context Window Usage:

Retrieved chunks (3000 tokens):
→ Mentions: Rate limit exists
→ States: 1000 per hour
→ Missing: Enforcement behavior

LLM trained on internet data:
→ "Knows" typical API behaviors
→ Generalizes from training
→ Fills gaps with plausible patterns

Result: Blends context + training knowledge
→ Cannot distinguish what's in context vs learned
→ Generates "typical" API behavior

Pattern Completion Bias:

LLM sees pattern: "rate limit → enforcement"
Training data common pattern:
→ Rate limits typically have retry delays
→ Often 60-second cooldowns
→ Some APIs ban after violations

Generates archetypal response:
→ Even if specific API doesn't work that way
→ "Sounds right" but isn't factually correct

Instruction Following vs Grounding

Tension between creativity and accuracy:

System Prompt Dilemma:

Instruction: "Answer based only on provided context"

But LLM also trained to:
→ Be helpful and complete
→ Answer user's actual question
→ Provide useful information

User asks: "What happens if I exceed limit?"
Context: Silent on this

LLM choice:
A) Say "I don't know" (honest but unhelpful)
B) Infer typical behavior (helpful but risky)

Often chooses B → hallucination

The Helpful Assistant Problem:

LLM fine-tuned to be helpful:
→ Avoid saying "I don't know"
→ Provide complete answers
→ Be conversational

Conflicts with RAG goal:
→ Only cite sources
→ Admit gaps in knowledge
→ Stay strictly grounded

Training objectives misaligned with RAG requirements

Confidence Calibration Failure

LLMs don't know what they don't know:

Equal Confidence for All Outputs:

Response based on context: "The limit is 1000/hour"
Response invented: "Suspension lasts 15 minutes"

Both generated with same confidence:
→ No internal uncertainty signal
→ User cannot distinguish
→ Both sound equally authoritative

LLM lacks epistemic awareness
→ Doesn't track source of information
→ Cannot flag invented content

The Plausibility Trap:

Hallucinated content is often plausible:
→ Follows typical API patterns
→ Uses correct terminology
→ Grammatically coherent
→ Logically consistent

Harder to detect than random nonsense:
→ "15-minute suspension" sounds reasonable
→ "Three strikes policy" is common pattern
→ User may accept as fact

Context Length Limitations

Retrieved context may be insufficient:

Incomplete Information:

Query: "Complete setup procedure"
Retrieved (partial):
→ Step 1: Install package
→ Step 3: Configure settings

Missing: Step 2

LLM fills gap:
→ Invents "Step 2: Initialize database"
→ Sounds logical
→ May be wrong

Should instead: Acknowledge missing steps

Contradictory Sources:

Chunk A: "Premium plan includes 10 users"
Chunk B: "Premium supports up to 5 users"

LLM must reconcile:
→ May choose one arbitrarily
→ May average: "Premium offers 5-10 users"
→ May invent: "Premium basic has 5, Premium plus has 10"

Adding details not in either source

Mitigation Strategies

1. Aggressive System Prompting:

Prompt: "ONLY answer using the provided context. If the 
context doesn't contain the answer, respond: 'I don't 
have that information in my knowledge base.' Never make 
up information."

Helps but doesn't eliminate hallucination
→ LLM still fills gaps unconsciously
→ Instruction adherence ~85%, not 100%

2. Citation Requirements:

Prompt: "For each claim, cite the source chunk. Format:
[Source: chunk_id]"

Forces grounding:
→ Cannot cite invented facts
→ Makes hallucination visible
→ User can verify claims

Example: "Rate limit is 1000/hour [Source: api_docs_chunk_5]"

3. Confidence Scoring:

Ask LLM: "On scale 1-5, how confident are you this 
information is in the provided context?"

Response: "Based on context, rate limit is 1000/hour 
(confidence: 5). Enforcement behavior: likely temporary 
suspension (confidence: 2 - inferred, not stated)."

Makes uncertainty explicit

4. Two-Stage Generation:

Stage 1: Extract relevant facts from context
→ LLM outputs: "Rate limit: 1000/hour. Enforcement: not specified"

Stage 2: Answer using only extracted facts
→ "The rate limit is 1000 requests per hour. The enforcement 
mechanism is not documented in the provided materials."

Structured pipeline reduces hallucination

---

How to Solve

Use strict system prompts requiring citation + implement confidence scoring + apply two-stage extraction-then-answer pipeline + penalize hallucination in prompt engineering + use models fine-tuned for RAG (instruction-following models). See Hallucination Prevention.