# Hallucination in Responses ## The Problem LLMs generate confident but factually incorrect information not present in retrieved context, leading to wrong answers despite having correct source material. ### Symptoms * ❌ AI invents facts not in retrieved documents * ❌ Confident responses with fabricated details * ❌ Mixing real and made-up information * ❌ Cannot cite source for invented claims * ❌ Plausible-sounding but wrong answers ### Real-World Example ``` Retrieved context: "API rate limit is 1000 requests per hour" User query: "What happens if I exceed the rate limit?" AI response: "If you exceed the rate limit of 1000 requests per hour, your account will be temporarily suspended for 15 minutes and you'll receive a 429 error. After three violations, your API key will be permanently revoked." Problem: Context only mentions the limit → "15 minutes suspension" - INVENTED → "three violations" policy - INVENTED → "permanent revocation" - INVENTED Only "1000 requests/hour" and "429 error" might be accurate ``` *** ## Deep Technical Analysis ### Retrieval-Generation Gap LLM operates beyond retrieved context: **Context Window Usage:** ``` Retrieved chunks (3000 tokens): → Mentions: Rate limit exists → States: 1000 per hour → Missing: Enforcement behavior LLM trained on internet data: → "Knows" typical API behaviors → Generalizes from training → Fills gaps with plausible patterns Result: Blends context + training knowledge → Cannot distinguish what's in context vs learned → Generates "typical" API behavior ``` **Pattern Completion Bias:** ``` LLM sees pattern: "rate limit → enforcement" Training data common pattern: → Rate limits typically have retry delays → Often 60-second cooldowns → Some APIs ban after violations Generates archetypal response: → Even if specific API doesn't work that way → "Sounds right" but isn't factually correct ``` ### Instruction Following vs Grounding Tension between creativity and accuracy: **System Prompt Dilemma:** ``` Instruction: "Answer based only on provided context" But LLM also trained to: → Be helpful and complete → Answer user's actual question → Provide useful information User asks: "What happens if I exceed limit?" Context: Silent on this LLM choice: A) Say "I don't know" (honest but unhelpful) B) Infer typical behavior (helpful but risky) Often chooses B → hallucination ``` **The Helpful Assistant Problem:** ``` LLM fine-tuned to be helpful: → Avoid saying "I don't know" → Provide complete answers → Be conversational Conflicts with RAG goal: → Only cite sources → Admit gaps in knowledge → Stay strictly grounded Training objectives misaligned with RAG requirements ``` ### Confidence Calibration Failure LLMs don't know what they don't know: **Equal Confidence for All Outputs:** ``` Response based on context: "The limit is 1000/hour" Response invented: "Suspension lasts 15 minutes" Both generated with same confidence: → No internal uncertainty signal → User cannot distinguish → Both sound equally authoritative LLM lacks epistemic awareness → Doesn't track source of information → Cannot flag invented content ``` **The Plausibility Trap:** ``` Hallucinated content is often plausible: → Follows typical API patterns → Uses correct terminology → Grammatically coherent → Logically consistent Harder to detect than random nonsense: → "15-minute suspension" sounds reasonable → "Three strikes policy" is common pattern → User may accept as fact ``` ### Context Length Limitations Retrieved context may be insufficient: **Incomplete Information:** ``` Query: "Complete setup procedure" Retrieved (partial): → Step 1: Install package → Step 3: Configure settings Missing: Step 2 LLM fills gap: → Invents "Step 2: Initialize database" → Sounds logical → May be wrong Should instead: Acknowledge missing steps ``` **Contradictory Sources:** ``` Chunk A: "Premium plan includes 10 users" Chunk B: "Premium supports up to 5 users" LLM must reconcile: → May choose one arbitrarily → May average: "Premium offers 5-10 users" → May invent: "Premium basic has 5, Premium plus has 10" Adding details not in either source ``` ### Mitigation Strategies **1. Aggressive System Prompting:** ``` Prompt: "ONLY answer using the provided context. If the context doesn't contain the answer, respond: 'I don't have that information in my knowledge base.' Never make up information." Helps but doesn't eliminate hallucination → LLM still fills gaps unconsciously → Instruction adherence ~85%, not 100% ``` **2. Citation Requirements:** ``` Prompt: "For each claim, cite the source chunk. Format: [Source: chunk_id]" Forces grounding: → Cannot cite invented facts → Makes hallucination visible → User can verify claims Example: "Rate limit is 1000/hour [Source: api_docs_chunk_5]" ``` **3. Confidence Scoring:** ``` Ask LLM: "On scale 1-5, how confident are you this information is in the provided context?" Response: "Based on context, rate limit is 1000/hour (confidence: 5). Enforcement behavior: likely temporary suspension (confidence: 2 - inferred, not stated)." Makes uncertainty explicit ``` **4. Two-Stage Generation:** ``` Stage 1: Extract relevant facts from context → LLM outputs: "Rate limit: 1000/hour. Enforcement: not specified" Stage 2: Answer using only extracted facts → "The rate limit is 1000 requests per hour. The enforcement mechanism is not documented in the provided materials." Structured pipeline reduces hallucination ``` *** ## How to Solve **Use strict system prompts requiring citation + implement confidence scoring + apply two-stage extraction-then-answer pipeline + penalize hallucination in prompt engineering + use models fine-tuned for RAG (instruction-following models).** See [Hallucination Prevention](/rag-scenarios-and-solutions/llm/hallucination-deep.md). --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://help.twig.so/rag-scenarios-and-solutions/llm/hallucination-deep.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.