Module 4.4: Question Transformations & Search Expansion

Curriculum Alignment: docs/plan/04_phase4_context_engineering_agentic_rag.md
Topic Scope: Rewrite-Retrieve-Read, Step-Back Queries, HyDE (Hypothetical Document Embeddings), Reciprocal Rank Fusion (RRF)
Level: Advanced AI Engineering / Architecture

1. Why Raw User Queries Fail Retrieval

In production search systems, user queries are frequently poorly formulated for vector search:

Conversational Ambiguity: "How do I fix that bug from yesterday?"
Keyword Under-specification: "Auth failure" (lacks protocol, service, or environment keywords).
Asymmetric Representation: A user query is an interrogation, while the target knowledge base contains affirmative documentation.

Question Transformations use an intermediate LLM step to reformulate, expand, or abstract the user query before querying the vector store:

text

┌────────────────────────────────────────────────────────────────────────┐
│ THE 4 CANONICAL QUESTION TRANSFORMATION PATTERNS                       │
│ (AI Agents and Applications, Chapter 9: Question Transformations)      │
├──────────────────────────┬─────────────────────────────────────────────┤
│ TECHNIQUE                │ CORE MECHANISM & PURPOSE                    │
├──────────────────────────┼─────────────────────────────────────────────┤
│ 1. Rewrite-Retrieve-Read │ Re-articulates vague queries with domain    │
│    (Ch. 9.1, p. 233)     │ keywords for crisp semantic matching.       │
├──────────────────────────┼─────────────────────────────────────────────┤
│ 2. Multi-Query & RRF     │ Generates 3–5 perspective queries and merges│
│    (Ch. 9.2, p. 240)     │ results via Reciprocal Rank Fusion.         │
├──────────────────────────┼─────────────────────────────────────────────┤
│ 3. Step-Back Queries     │ Generates a higher-level first-principles   │
│    (Ch. 9.3, p. 245)     │ query to retrieve foundational concepts.    │
├──────────────────────────┼─────────────────────────────────────────────┤
│ 4. HyDE Embeddings       │ Generates a hypothetical answer document    │
│    (Ch. 9.4, p. 249)     │ and embeds the answer rather than question. │
└──────────────────────────┴─────────────────────────────────────────────┘

2. Deep Dive into the 4 Transformation Patterns

1. Rewrite-Retrieve-Read (Ch. 9.1, p. 233)

Before sending the query to ChromaDB, a rewriter chain injects technical context, expands acronyms, and removes conversational noise:

\text{Query: "Why is login broken?"} \longrightarrow \text{Rewritten: "OAuth2 authorization code flow invalid_grant token expiration troubleshooting"}

2. Multi-Query & Reciprocal Rank Fusion (RRF) (Ch. 9.2, p. 240)

Different phrasing explores different regions of embedding space. The multi-query chain generates 3–5 distinct variations of the query, executes parallel vector searches, and merges the ranked lists using Reciprocal Rank Fusion (RRF):

RRF_Score (d) = \sum_{q \in Q} \frac{1}{k + rank (d, q)} (k \approx 60)

Documents that appear near the top across multiple query variants receive the highest composite score, neutralizing query formulation bias.

3. Step-Back Question Generation (Ch. 9.3, p. 245)

When a query is bogged down in specific parameters, the step-back chain derives the overarching concept:

Specific Query: "Why does service X crash when handling 10,000 concurrent Redis connections?"
Step-Back Query: "What are the connection pooling and file descriptor limits in Redis architecture?" Retrieving both the specific and step-back documents gives the generator both the low-level facts and the high-level principles needed for a complete diagnosis.

4. Hypothetical Document Embeddings (HyDE) (Ch. 9.4, p. 249)

The Asymmetry Problem: Questions and answers occupy different semantic spaces. Embedding a question ("How do I configure Kafka partitions?") often fails to match the actual answer passage ("Partitions are configured using num.partitions in server.properties...").
The HyDE Mechanism: An LLM generates a hypothetical answer passage based on its pre-trained weights. Even if the hypothetical answer contains hallucinated specifics, its vector embedding captures the exact semantic, syntactic, and grammatical structure of an answer, resulting in significantly higher vector alignment with real ground-truth documents.

3. Production Python Implementation: Question Transformation Pipeline

python

"""
Multi-Query and HyDE Transformation Engine
Demonstrates question expansion and Reciprocal Rank Fusion (RRF).
"""
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_openai import ChatOpenAI
from langchain_community.vectorstores import Chroma

model = ChatOpenAI(model="gpt-4o-mini", temperature=0.0)

# 1. Multi-Query Generator Chain
multi_query_prompt = ChatPromptTemplate.from_template("""You are an AI search query optimization assistant.
Generate 3 distinct search query variations for the user question to search an enterprise knowledge base.
Provide each query on a new line without numbering.

Original Question: {question}
""")
multi_query_chain = multi_query_prompt | model | StrOutputParser()

# 2. HyDE Generator Chain (Hypothetical Document Embeddings)
hyde_prompt = ChatPromptTemplate.from_template("""You are an expert technical documentation writer.
Write a concise, 1-paragraph hypothetical answer to the question below.
Do not say "I don't know". Write as if this were an excerpt from the official system documentation.

Question: {question}
""")
hyde_chain = hyde_prompt | model | StrOutputParser()

def reciprocal_rank_fusion(results_by_query: list[list[str]], k: int = 60) -> list[tuple[str, float]]:
    """Merges multiple ranked document lists into a single ranked list using RRF."""
    scores: dict[str, float] = {}
    
    for doc_list in results_by_query:
        for rank, doc in enumerate(doc_list):
            if doc not in scores:
                scores[doc] = 0.0
            scores[doc] += 1.0 / (k + rank + 1)
            
    # Sort descending by composite RRF score
    return sorted(scores.items(), key=lambda x: x[1], reverse=True)

Conceptual Mindmap: Question Transformations

4. Curated Reading & Canonical References

Resource	Canonical Reference & Link	Specific Focus Areas
Primary Curriculum Book	AI Agents and Applications (Google Drive)	Part 3, Chapter 9 (pp. 232–253): "Question transformations" — Section 9.1 (Rewrite-Retrieve-Read), Section 9.2 (Multi-query), Section 9.3 (Step-back questions), Section 9.4 (HyDE).
Foundational HyDE Paper	Precise Zero-Shot Dense Retrieval without Relevance Labels (Gao et al., 2022)	Hypothetical document embedding mechanics and dense semantic alignment.
Step-Back Prompting	Take a Step Back: Evoking Reasoning via Abstraction (Zheng et al., 2023)	Question abstraction for dual-context retrieval in scientific and engineering queries.

5. Active Recall (Module 4.4 Flashcards)

Question TransformationsClick or press Space to flip ↺

Why does Hypothetical Document Embeddings (HyDE) improve dense vector retrieval accuracy?

Question Transformations • AnswerClick to flip back ↻

Queries and documents are inherently asymmetric: queries are short questions; documents are detailed answers. Because vector embeddings reflect syntactic and structural style, a question vector often has low cosine similarity to the ideal answer. HyDE generates a hypothetical answer first, embedding an 'answer-like' text that aligns closely in vector space with real knowledge base passages.

💡 Architect Takeaway: HyDE transforms question-to-document search into document-to-document similarity matching.

Rank FusionClick or press Space to flip ↺

How does Reciprocal Rank Fusion (RRF) merge divergent multi-query search results?

Rank Fusion • AnswerClick to flip back ↻

RRF assigns each document a score equal to the sum of 1 / (k + rank) across all query variants. Documents that consistently appear in the top results across multiple query phrasings achieve the highest composite score, neutralizing individual query bias.

💡 Architect Takeaway: Use RRF to combine sparse BM25 and dense vector search results robustly.

6. Hands-on Engineering Exercises

Exercise 4.4: The Rewrite-Retrieve-Read Comparison

Goal: Observe search expansion gains empirically.
Task: Take a vague query ("Why did the batch job fail?") and compare:
1. Direct retrieval on raw query vs.
2. Rewrite-Retrieve-Read with expanded system keywords ("Celery worker Redis OOM killed SIGKILL status code 137").
Verification: Verify that the rewritten query retrieves the exact incident post-mortem that the raw query missed.

Module 4.4: Question Transformations & Search Expansion ​

1. Why Raw User Queries Fail Retrieval ​

2. Deep Dive into the 4 Transformation Patterns ​

1. Rewrite-Retrieve-Read (Ch. 9.1, p. 233) ​

2. Multi-Query & Reciprocal Rank Fusion (RRF) (Ch. 9.2, p. 240) ​

3. Step-Back Question Generation (Ch. 9.3, p. 245) ​

4. Hypothetical Document Embeddings (HyDE) (Ch. 9.4, p. 249) ​

3. Production Python Implementation: Question Transformation Pipeline ​

Conceptual Mindmap: Question Transformations ​

4. Curated Reading & Canonical References ​

5. Active Recall (Module 4.4 Flashcards) ​