Module 4.5: The RAG Triad & Evaluation Engineering

Curriculum Alignment: docs/plan/04_phase4_context_engineering_agentic_rag.md
Topic Scope: The RAG Triad (Faithfulness, Context Precision, Answer Relevance), Ragas, DeepEval, Offline Pytest Suites
Level: Advanced AI Engineering / Architecture

1. Moving Beyond "Vibe Checks": Scientific Evaluation

In early RAG experiments, developers typically test a handful of queries manually, look at the answer, and conclude "it looks good". In production, this "vibe check" approach is disastrous:

A slight change to chunk size or embedding model can silently degrade retrieval precision across thousands of edge cases.
Prompt tweaks intended to improve formatting can trigger subtle hallucinations that evade manual inspection.

To engineer reliable RAG systems, we apply quantitative metric frameworks (Ragas and DeepEval) anchored on The RAG Triad:

text

┌────────────────────────────────────────────────────────────────────────┐
│ THE RAG TRIAD EVALUATION FRAMEWORK                                     │
├────────────────────┬────────────────────┬──────────────────────────────┤
│ METRIC             │ EVALUATION TARGET  │ FORMULA / CONCEPT            │
├────────────────────┼────────────────────┼──────────────────────────────┤
│ 1. Context         │ Retriever Quality  │ What % of retrieved chunks   │
│    Precision       │ (Signal vs Noise)  │ are actually relevant?       │
├────────────────────┼────────────────────┼──────────────────────────────┤
│ 2. Answer          │ Generator Grounding│ Is every claim in the answer │
│    Faithfulness    │ (Zero Hallucination│ directly supported by the    │
│                    │ Rate)              │ retrieved context?           │
├────────────────────┼────────────────────┼──────────────────────────────┤
│ 3. Answer          │ Generator Utility  │ Does the answer directly and │
│    Relevance       │ (Query Alignment)  │ completely address the user's│
│                    │                    │ question?                    │
└────────────────────┴────────────────────┴──────────────────────────────┘

2. Mathematical Formulations of the Triad

1. Answer Faithfulness (Grounding Score)

Evaluates whether the generator stayed strictly within the facts present in the retrieved context:

Faithfulness = \frac{| Number of Claims in Answer Supported by Context |}{| Total Number of Claims in Answer |}

A score of $1.0$ guarantees zero hallucination. If the model introduces external knowledge or speculation not present in the context, the score drops.

2. Context Precision

Measures whether the relevant information was ranked at the very top of the retrieved chunks (rewarding high rank order to mitigate "Lost in the Middle"):

Context Precision@K = \frac{\sum_{k = 1}^{K} Precision@k \times relevance (k)}{Total Relevant Documents}

3. Answer Relevance

Measures whether the answer answered what was actually asked. Evaluated by prompting an evaluator model to generate questions from the generated answer and computing semantic cosine similarity between the generated questions and the original user question:

Answer Relevance = \frac{1}{N} \sum_{i = 1}^{N} \cos (e_{q}, e_{q_{i}})

3. Automated Offline Evaluation with `pytest` and `Ragas`

In production CI/CD pipelines, RAG systems must be guarded by automated regression test suites. Before merging a new chunking strategy, re-ranker, or system prompt, automated tests score the pipeline against a versioned gold-standard dataset:

python

"""
Automated RAG Triad Evaluation Suite using Pytest & Ragas
Validates that pipeline changes do not regress Faithfulness below 0.90.
"""
import pytest
from pydantic import BaseModel
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevance, context_precision
from datasets import Dataset

# Versioned Gold-Standard Evaluation Dataset
EVAL_DATASET = {
    "question": [
        "What is the maximum allowed replication lag for the payment database?",
        "How is disaster recovery failover initiated in us-west-2?"
    ],
    "ground_truth": [
        "The maximum allowed replication lag is 500 milliseconds before alerting.",
        "Disaster recovery failover is initiated via the runbook script 'dr-failover.sh' with senior SRE authorization."
    ]
}

def execute_rag_pipeline(question: str) -> dict[str, Any]:
    """Runs the production RAG pipeline and returns answer + contexts."""
    # Simulated return from production pipeline
    return {
        "answer": "The maximum allowed replication lag is 500ms before P1 alerts fire.",
        "contexts": [
            "Section 4.2 Database SLAs: The maximum allowed replication lag for the payment cluster is 500 milliseconds."
        ]
    }

@pytest.mark.rag_eval
def test_rag_pipeline_quality_standards():
    """CI/CD Gate: Verifies that RAG pipeline meets minimum production thresholds."""
    questions = EVAL_DATASET["question"]
    ground_truths = EVAL_DATASET["ground_truth"]
    
    answers = []
    contexts = []
    
    for q in questions:
        res = execute_rag_pipeline(q)
        answers.append(res["answer"])
        contexts.append(res["contexts"])
        
    dataset = Dataset.from_dict({
        "question": questions,
        "answer": answers,
        "contexts": contexts,
        "ground_truth": ground_truths
    })
    
    # Run quantitative evaluation across the RAG Triad
    score = evaluate(
        dataset=dataset,
        metrics=[faithfulness, answer_relevance, context_precision]
    )
    
    # Assert Strict Production Quality Gates
    assert score["faithfulness"] >= 0.90, f"Faithfulness regressed: {score['faithfulness']}"
    assert score["answer_relevance"] >= 0.85, f"Relevance regressed: {score['answer_relevance']}"
    assert score["context_precision"] >= 0.80, f"Precision regressed: {score['context_precision']}"

Conceptual Mindmap: RAG Triad & Evaluation

4. Curated Reading & Canonical References

Resource	Canonical Reference & Link	Specific Focus Areas
Primary Curriculum Book	AI Agents and Applications (Google Drive)	Part 4, Chapter 14, Section 14.3.4 (pp. 374–375): "Evaluation of AI agents and applications" — benchmarking agent accuracy and regression testing.
Foundational Metric Paper	Ragas: Automated Evaluation of Retrieval Augmented Generation (Es et al., 2023)	Mathematical formulations for Faithfulness, Answer Relevance, and Context Precision.
Evaluation Framework	Ragas GitHub Repository	Test set synthesis, metric definitions, and LangSmith/LangFuse integration.

5. Active Recall (Module 4.5 Flashcards)

RAG EvaluationClick or press Space to flip ↺

What does Answer Faithfulness measure, and why is it the #1 safety metric in enterprise RAG?

RAG Evaluation • AnswerClick to flip back ↻

Faithfulness measures the proportion of claims made in the generated answer that can be directly verified against the retrieved context passages. A faithfulness score < 1.0 indicates that the model hallucinated facts or brought in unverified external training priors.

💡 Architect Takeaway: Enforce a strict >= 0.90 Faithfulness threshold before releasing RAG pipelines to production.

CI/CD TestingClick or press Space to flip ↺

How do automated offline pytest suites prevent RAG regression during model upgrades?

CI/CD Testing • AnswerClick to flip back ↻

By running a versioned gold-standard dataset of representative questions and ground-truth answers through the pipeline, scoring results with Ragas, and failing the CI build if Faithfulness, Precision, or Relevance drop below baseline thresholds.

💡 Architect Takeaway: Automated regression tests ensure that prompt tweaks or model swaps never silently break factual accuracy.

6. Hands-on Engineering Exercises

Exercise 4.5: The Ragas Metric Benchmark Drill

Goal: Build the foundation for your Advanced RAG Dashboard deliverable.
Task: Create a curated test set of 5 questions with verified ground-truth answers.
Verification:
- Run the evaluation script using Ragas.
- Generate a Markdown evaluation summary reporting Faithfulness, Answer Relevance, and Context Precision scores.
- Demonstrate that the pipeline achieves $\geq 90 %$ Faithfulness.

Module 4.5: The RAG Triad & Evaluation Engineering ​

1. Moving Beyond "Vibe Checks": Scientific Evaluation ​

2. Mathematical Formulations of the Triad ​

1. Answer Faithfulness (Grounding Score) ​

2. Context Precision ​

3. Answer Relevance ​

3. Automated Offline Evaluation with pytest and Ragas ​

Conceptual Mindmap: RAG Triad & Evaluation ​

4. Curated Reading & Canonical References ​

5. Active Recall (Module 4.5 Flashcards) ​