5.5 Trace-Driven Failure Analysis & The Evaluation Data Flywheel

Canonical Curriculum Reference: docs/plan/05_phase5_harness_engineering_mcp.md
Textbook Reference: AI Agents and Applications (Manning Publications), Chapter 11, Section 11.9 ("Observing and debugging with LangSmith", pp. 317–319) & Chapter 14, Section 14.3.4 (pp. 375–376).

---

🔬 1. Trace-Driven Failure Analysis: The 4 Core Taxonomies

In distributed systems, an HTTP 500 error immediately pinpoints a crash, while an HTTP 200 indicates success. In Generative AI systems, HTTP 200 OK tells you almost nothing about application correctness. An LLM can return HTTP 200 while hallucinating a solution, calling destructive tools erroneously, or completely failing the user's objective.

To diagnose and remediate agentic systems systematically, senior engineers classify failures by inspecting the multi-step trace in observability platforms like LangFuse:

+-----------------------------------------------------------------------+
|                    THE 4 AGENTIC FAILURE TAXONOMIES                   |
+-----------------------------------------------------------------------+
| 1. REASONING FAILURE                                                  |
|    - Symptoms: Context was complete; tool outputs were valid.         |
|    - Root Cause: Flawed deductive step, arithmetic error, or          |
|      inability to synthesize a logical path.                          |
|    - Remedy: Switch to CoT/ToT reasoning, few-shot steering, or SLM. |
+-----------------------------------------------------------------------+
| 2. TOOL INVOCATION FAILURE                                            |
|    - Symptoms: Model emitted invalid JSON, wrong arguments, or        |
|      invoked non-existent tools.                                      |
|    - Root Cause: Ambiguous schema description or missing validation.  |
|    - Remedy: Strict Pydantic schemas via .with_structured_output().   |
+-----------------------------------------------------------------------+
| 3. CONTEXT INSUFFICIENCY                                              |
|    - Symptoms: Model made invalid assumptions or asked for info       |
|      already present in the repository.                               |
|    - Root Cause: Missing retrieval, bad chunking, or context trough.  |
|    - Remedy: Advanced RAG, Small-to-Big retrieval, LocalContext.      |
+-----------------------------------------------------------------------+
| 4. HALLUCINATION OF COMPLETION                                        |
|    - Symptoms: Model claimed task was complete despite failing tests. |
|    - Root Cause: Lack of deterministic execution verification.        |
|    - Remedy: Build-Verify-Fix loop & PreCompletionChecklist.          |
+-----------------------------------------------------------------------+

---

🔄 2. The Data Flywheel: Converting Traces to Regression Tests

The most powerful competitive advantage in AI engineering is the Data Flywheel: the systematic process of turning production failures into permanent, automated regression tests.

[ Production Incident ]
          |
          v
[ Capture LangFuse Trace ] ---> Tag: { failure_taxonomy: "TOOL_INVOCATION" }
          |
          v
[ Extract Input & Expected Output ]
          |
          v
[ Synthesize Automated Pytest Case ] ---> Saved to `tests/regression/test_trace_*.py`
          |
          v
[ CI/CD Gate Activated ] ---> Any future regression is blocked before deployment

When an agent fails in production:

1. Trace Export: The root trace ID is flagged in LangFuse with the specific taxonomy category.
2. Deterministic Serialization: The user's input prompt, tool definitions, and ground-truth validation criteria are extracted.
3. Automated Test Generation: The harness generates a reproducible pytest test case that replays the exact input against the agent pipeline, asserting that the failure mode is eradicated.

---

💻 3. Production Implementation: The Data Flywheel Pipeline

from enum import StrEnum
from pathlib import Path
from typing import Any, Final
from pydantic import BaseModel, ConfigDict, Field


class FailureTaxonomy(StrEnum):
    REASONING_FAILURE = "REASONING_FAILURE"
    TOOL_INVOCATION_FAILURE = "TOOL_INVOCATION_FAILURE"
    CONTEXT_INSUFFICIENCY = "CONTEXT_INSUFFICIENCY"
    HALLUCINATION_OF_COMPLETION = "HALLUCINATION_OF_COMPLETION"


class CapturedTraceRecord(BaseModel):
    """Normalized production failure record extracted from LangFuse."""
    model_config = ConfigDict(frozen=True, extra="forbid")
    
    trace_id: str
    taxonomy: FailureTaxonomy
    user_prompt: str
    agent_trajectory: list[dict[str, Any]]
    expected_outcome_assertion: str
    environment_metadata: dict[str, str] = Field(default_factory=dict)


class DataFlywheelGenerator:
    """Transforms production trace failures into versioned pytest regression tests."""

    def __init__(self, regression_test_dir: Path) -> None:
        self._output_dir: Final[Path] = regression_test_dir.resolve()
        self._output_dir.mkdir(parents=True, exist_ok=True)

    def generate_pytest_case(self, record: CapturedTraceRecord) -> Path:
        """Synthesizes an executable pytest module from a failure trace."""
        test_name = f"test_regression_{record.trace_id.replace('-', '_')}"
        file_path = self._output_dir / f"{test_name}.py"
        
        content = f'''"""
Automated Regression Test derived from Production Failure Trace.
Trace ID: {record.trace_id}
Taxonomy: {record.taxonomy}
"""
import pytest
from pydantic import BaseModel

def {test_name}():
    # 1. Arrange: Replay original user prompt and context
    prompt = {repr(record.user_prompt)}
    
    # 2. Act: Execute candidate agent pipeline
    # In production, call agent.invoke({{"prompt": prompt}})
    simulated_result = "verified_output"
    
    # 3. Assert: Verify the historical failure mode is permanently mitigated
    assert simulated_result is not None, "Pipeline returned empty completion"
    # User-defined verification assertion
    {record.expected_outcome_assertion}
'''
        file_path.write_text(content.strip())
        return file_path

---

🧪 4. Automated Verification with Pytest

import pytest
from pathlib import Path
from tempfile import TemporaryDirectory

def test_data_flywheel_generates_valid_pytest_module():
    with TemporaryDirectory() as tmp_dir:
        out_dir = Path(tmp_dir)
        generator = DataFlywheelGenerator(regression_test_dir=out_dir)
        
        record = CapturedTraceRecord(
            trace_id="trace-abc-123",
            taxonomy=FailureTaxonomy.HALLUCINATION_OF_COMPLETION,
            user_prompt="Refactor database models and ensure migrations pass.",
            agent_trajectory=[{"step": 1, "action": "fake_completion"}],
            expected_outcome_assertion="assert 'verified_output' in simulated_result"
        )
        
        test_file = generator.generate_pytest_case(record)
        assert test_file.exists()
        assert test_file.name == "test_regression_trace_abc_123.py"
        
        file_content = test_file.read_text()
        assert "HALLUCINATION_OF_COMPLETION" in file_content
        assert "assert 'verified_output' in simulated_result" in file_content

---

📊 Long-Term Engineering Value

Establishing a trace-driven data flywheel ensures:

1. Zero Recurring Incidents: Once a failure is captured and converted into a regression test, that exact failure mode can never silently reappear in future prompt iterations.
2. Defensible Model Upgrades: When evaluating whether to upgrade from GPT-4o-mini to Claude 3.5 Sonnet or a fine-tuned SLM, running the full suite of flywheel tests provides empirical pass-rate proof.