Source Attribution Tracking

The Problem

Cannot verify which sources the AI actually used for its response, making it impossible to validate accuracy or trace misinformation.

Symptoms

• ❌ AI cites sources but unclear which claims from which source

• ❌ Cannot verify if citation accurate

• ❌ Multiple sources cited but unclear contribution

• ❌ Hallucination detection impossible

• ❌ No source-to-claim mapping

Real-World Example

AI response:
"The API rate limit is 1000 requests per hour. Premium users get priority support.
Billing is monthly. [Sources: doc_123, doc_456, doc_789]"

Questions:
→ Which source says "1000 requests/hour"?
→ Which source mentions "priority support"?
→ What does doc_789 contribute?

No granular attribution:
→ Cannot verify each claim
→ Cannot detect if "priority support" hallucinated
→ Bulk citation unhelpful

---

Deep Technical Analysis

Granular Citation Tracking

Claim-Level Attribution:

{
  "response": "The API rate limit is 1000 requests per hour...",
  "claims": [
    {
      "text": "rate limit is 1000 requests per hour",
      "source": {
        "chunk_id": "doc_123_chunk_5",
        "document": "API Documentation v2.0",
        "snippet": "...enforces 1000 req/hour limit...",
        "confidence": 0.95
      }
    },
    {
      "text": "Premium users get priority support",
      "source": {
        "chunk_id": "doc_456_chunk_12",
        "document": "Premium Features Guide",
        "snippet": "...priority support for Premium tier...",
        "confidence": 0.88
      }
    }
  ]
}

Unsupported Claims Detection:

Claim: "Billing is monthly"

Check retrieved chunks:
→ No chunk mentions "monthly billing"
→ Flag: Potential hallucination

Log:
{
  "claim": "Billing is monthly",
  "source": null,
  "flag": "unsupported_claim",
  "confidence": 0.0
}

Alert: Manual review needed

Source Usage Analytics

Source Utilization:

Retrieved 10 chunks:
→ Chunk 1: Used (cited in response)
→ Chunk 2: Used
→ Chunk 3: Not used
→ Chunk 4: Used
→ ...

Utilization: 6/10 = 60%

Analysis:
→ 4 chunks retrieved but unused (why?)
→ Wasted retrieval
→ Could reduce K

Chunk Contribution:

Track how much each chunk contributed:
{
  "chunk_id": "doc_123_chunk_5",
  "contribution": {
    "claims": 3,
    "tokens_used": 120,
    "importance": "high"
  }
}

Identify most valuable chunks:
→ Boost in ranking
→ Prioritize similar content

Citation Formats

Inline Citations:

Format response with inline sources:
"The API rate limit is 1000 requests per hour [1]. Premium users
get priority support [2]."

References:
[1] API Documentation v2.0, Section 3.2
[2] Premium Features Guide, page 5

Clear claim-to-source mapping

Hover Citations (UI):

User hovers over "1000 requests per hour":
→ Popup shows: Source chunk with highlighted text

Interactive verification:
→ User can check source
→ Builds trust

Hallucination Detection

Source Grounding Check:

For each claim in response:
1. Extract claim
2. Search retrieved chunks for support
3. If found: Calculate semantic similarity
4. If similarity < 0.70: Flag as potential hallucination

Example:
Claim: "Premium plan costs $500/month"
Best matching chunk: "Enterprise plan costs $500/month"
Similarity: 0.75 (high)
Issue: "Premium" vs "Enterprise" (different plans!)
Flag: Substitution error

Citation Validation:

AI claims: "According to doc_123: Rate limit is 1000/hour"

Verify:
→ Retrieve doc_123
→ Check if it actually says "1000/hour"
→ Semantic similarity: 0.92 (confirmed)

If mismatch:
→ Flag: Incorrect citation
→ Possible hallucination

Transparency Metrics

Source Confidence:

For each cited source:
→ Retrieval score: 0.85
→ Reranking score: 0.91
→ LLM confidence in using it: 0.88

Aggregate confidence: 0.88

Display to user:
"High confidence" (>0.80)
"Medium confidence" (0.60-0.80)
"Low confidence" (<0.60)

Citation Coverage:

Response has 5 claims:
→ 4 claims have citations
→ 1 claim uncited

Citation coverage: 80%

Target: >95% citation coverage
→ Minimize unsupported claims

---

How to Solve

Implement claim-level source attribution (map each claim to specific chunk) + track chunk utilization (which chunks actually used in response) + detect unsupported claims (no source found) + validate citations (check if source actually says what AI claims) + display inline citations in responses + monitor citation coverage percentage + flag low-confidence attributions + build source verification UI (hover to see chunk) + test attribution accuracy with eval set. See Source Attribution.