Strategy Selection Failures

The Problem

Agent chooses suboptimal RAG strategy (Redwood vs Cedar vs Cypress) for query, leading to poor results or inefficient processing.

Symptoms

• ❌ Simple query uses complex Cypress (slow + expensive)

• ❌ Complex query uses basic Redwood (incomplete answer)

• ❌ Strategy selection inconsistent

• ❌ Cannot predict which strategy agent will use

• ❌ Wrong strategy for user context

Real-World Example

Query: "What is the API rate limit?" (Simple factual)

Expected: Redwood (standard RAG, fast, cheap)
→ Retrieve chunks
→ Answer directly

Agent chose: Cypress (advanced agentic, slow, expensive)
→ Planning phase
→ Multi-step reasoning
→ Tool calls
→ Same answer, 5x latency, 3x cost

Strategy overkill for simple query

---

Deep Technical Analysis

Strategy Characteristics

Redwood (Standard RAG):

Best for:
→ Simple factual queries
→ Single-hop reasoning
→ Well-documented topics

Example: "What is X?"
→ Fast: 1-2 seconds
→ Cost: $0.01/query

Cedar (Context-Aware):

Best for:
→ User-specific queries
→ Permission-based retrieval
→ Personalized answers

Example: "What can I access?"
→ Needs user context
→ Moderate: 2-4 seconds
→ Cost: $0.02/query

Cypress (Advanced Agentic):

Best for:
→ Complex multi-step
→ Requires planning
→ Tool orchestration

Example: "Compare X, Y, Z and recommend"
→ Slow: 5-15 seconds
→ Cost: $0.05-0.10/query

Strategy Selection Logic

Query Complexity Detection:

Features:
→ Query length (tokens)
→ Question type (what, how, compare)
→ Named entities count
→ Conditional clauses

Simple: "What is X?" → Redwood
Medium: "How does X work with Y?" → Cedar
Complex: "If A then compare B vs C, considering D" → Cypress

Context Requirements:

Check if query needs:
→ User permissions: Use Cedar
→ Conversation history: Use Cedar
→ External tools: Use Cypress

Keywords:
→ "I", "my", "our" → User-specific → Cedar
→ "compare", "analyze", "recommend" → Complex → Cypress

Failure Modes

Under-Strategizing:

Complex query: "Explain how authentication, authorization, and audit
logging interact in our system"

Agent uses: Redwood (simple)
→ Retrieves authentication docs
→ Gives partial answer
→ Misses interaction analysis

Should use: Cypress
→ Break into sub-queries
→ Retrieve for each component
→ Synthesize relationships

Over-Strategizing:

Simple query: "API rate limit?"

Agent uses: Cypress
→ Planning: "Let me break this down..."
→ Sub-query 1: Rate limits
→ Sub-query 2: API documentation
→ Synthesis: Combine findings
→ 10 seconds

Redwood sufficient:
→ One retrieval
→ Direct answer
→ 2 seconds

Optimization Strategies

Classification Model:

Train classifier:
→ Input: Query text
→ Output: Strategy (Redwood / Cedar / Cypress)

Training data:
→ 1000 queries labeled by experts
→ Features: Length, complexity, keywords

Accuracy target: >85%

Rule-Based Heuristics:

Rules:
1. If query < 20 tokens AND simple question word → Redwood
2. If query contains "my", "I", "our" → Cedar
3. If query contains "compare", "analyze", "steps" → Cypress
4. Default: Redwood

Fast, interpretable

Adaptive Selection:

Start with Redwood:
→ If insufficient (confidence < 0.70) → Escalate to Cedar
→ If still insufficient → Escalate to Cypress

Progressive complexity
Minimize cost while ensuring quality

---

How to Solve

Implement query complexity classifier (ML or rule-based) + detect user-context requirements (keywords: my, I, our) + use Redwood by default, escalate if needed + monitor strategy selection accuracy + measure cost/latency per strategy + test strategy assignment on eval set + provide strategy override for power users + alert on frequent strategy escalations (indicates classifier issue). See Strategy Selection.