Phase 4 Lab Guide: Building the Deliverables

Curriculum Source: docs/plan/04_phase4_context_engineering_agentic_rag.md
Evaluation Skill: lab-evaluator

---

🎯 Required Deliverables Specification

When you are ready to write the implementation code for Phase 4, you will implement the following 4 deliverables:

---

1. The Memory-Enabled Travel Chatbot

• Objective: Build a conversational agent that leverages Episodic Memory to recall user style, constraints, and past preferences across multiple conversation threads using a shared user identifier / store.
• Requirements:
  • Implements the Three-Layer Memory Model:
    • Working Memory: Manages the immediate sliding-window context of the current session thread.
    • Episodic Memory: Extracts and stores declarative user facts, dietary restrictions, travel pacing, and past trip feedback into a persistent store (e.g. SQLite, JSON, or vector/key-value store).
    • Semantic Memory: References static knowledge (destination guides, visa rules, flight policies).
  • Demonstrates cross-conversation continuity:
    • Thread 1: User details dietary constraints ("Strictly vegan, prefer boutique hotels with quiet workspaces").
    • Thread 2 (New Session / Distinct Thread ID, same User ID): Agent proactively adapts recommendations without re-prompting.
  • Automated pytest test asserting that episodic facts are correctly retrieved and injected into prompt context across independent threads.

---

2. Advanced RAG Dashboard (>90% Faithfulness)

• Objective: Construct an Advanced RAG pipeline that achieves >90% Answer Faithfulness on a complex, multi-document domain dataset, verified with Ragas or DeepEval.
• Requirements:
  • Ingestion using intelligent chunking: ParentDocumentRetriever or MultiVectorRetriever (mapping sub-chunks or summary vectors to full parent sections).
  • Retrieval engine combining Dense Vector Search with Sparse BM25 via Reciprocal Rank Fusion (RRF).
  • Neural Re-Ranking stage utilizing a Cross-Encoder (e.g. Cohere Re-rank or BGE-Reranker) to filter top-k chunks into the model context.
  • Evaluation run executing a versioned evaluation dataset of at least 15 complex queries, outputting a formal Ragas evaluation report confirming:
    • $\text{Answer Faithfulness}\ge 0.90$
    • $\text{Context Precision}\ge 0.85$
    • $\text{Answer Relevance}\ge 0.85$

---

3. Hardened RAG Pipeline (Zero-Trust Security)

• Objective: Construct a production-grade ingestion and retrieval pipeline with active protection against Corpus Poisoning (BadRAG) and Embedding Inversion (Vec2Text attacks).
• Requirements:
  • Cryptographic Document Provenance:
    • Every ingested document chunk metadata includes author, ingestion_timestamp, document_hash, and an HMAC-SHA256 or RSA digital signature.
    • Retrieval pipeline validates signatures prior to context assembly. Any forged, tampered, or untrusted chunk is quarantined and dropped.
  • Embedding Inversion Defense:
    • Vector embeddings are protected via calibrated Gaussian noise perturbation ($\tilde{v}=\text{Normalize}(v+ϵ)$, $ϵ\sim \mathcal{N}(0,{\sigma }^2)$) or differential privacy transforms before external vector DB storage, preventing textual inversion.
  • Mandatory PII Scrubbing:
    • Ingestion pre-processors detect and redact sensitive PII (emails, phone numbers, SSNs, and API keys) before indexing.
  • Automated red-teaming tests proving that poisoned/unsigned documents are rejected and PII is never stored in vectors.

---

4. Quality Benchmarking Report (Naive vs. Agentic RAG)

• Objective: Conduct a comprehensive empirical benchmark demonstrating the quantifiable jump in accuracy and reliability from Naive RAG to Agentic RAG.
• Requirements:
  • Prepares a curated gold dataset of 20+ challenging multi-hop questions requiring document synthesis, ambiguity resolution, and temporal filtering.
  • Executes the dataset across two pipeline architectures:
    1. Naive RAG Baseline: Fixed chunk size (500 tokens), single-shot dense cosine retrieval, direct synthesis (typically 60–70% accuracy).
    2. Agentic RAG Architecture: Query routing, Rewrite-Retrieve-Read / HyDE, Self-Reflection loops, and Cross-Encoder re-ranking.
  • Delivers a structured evaluation report (benchmark_report.md or automated pytest output) detailing:
    • Comparative table of Faithfulness, Context Precision, and Answer Relevance.
    • Failure analysis: Documenting instances where Naive RAG hallucinated or fell into the "Lost in the Middle" trap, while Agentic RAG resolved the ambiguity.