Phase 5: Harness Engineering & Model Context Protocol (MCP)

Canonical Source of Trust: Google Doc Tab Phase 5
Training Program: [Huy Chau] Generative AI Training Plan

---

Harness Engineering & MCP Consumption

1. What to Learn

• The AI Harness Concept: Understanding that a harness is the system of tools, prompts, and execution flows built around a model to optimize for performance, token efficiency, and latency.
• Environment Onboarding: Using LocalContextMiddleware to help the agent "discover" its execution environment, such as mapping directory structures and finding tool installations (e.g., Python or Git).
• MCP Consumption: Utilizing the Model Context Protocol (MCP) to subscribe to remote tools without custom glue code
• Build-Verify-Fix Loops: Moving from superficial code review to aggressive self-verification, where the agent must run programmatic tests and compare results against the original specification before terminating.
• Deterministic Middleware Hooks: Implementing LoopDetectionMiddleware to break "doom loops" (repetitive edits to the same file) and PreCompletionChecklistMiddleware to force a final verification pass, reducing "hallucinations of completion”
• **Trace-Driven Failure Analysis: **
  • Distinguishing Reasoning Failures
  • Tool Invocation Failures
  • Context Insufficiency
  • Hallucination of Completion from a Trace
• **The Data Flywheel (Manual Prototype): **
  • Capturing a Failure Trace in LangFuse
  • Converting it into a Permanent Regression Test Case
• Operational Safety:
  • AIRateLimiter to enforce per-user request/token limits and daily cost caps to prevent unbounded consumption
  • The circuit breaker patterns and tool error classification

2. How to Learn

• Trace-Driven Debugging: Using LangFuse traces as the primary feedback signal to identify if an agent failed due to a reasoning error, a missing tool, or poor instructions.
• Automated Error Analysis: Implementing a "Trace Analyzer Skill" where a specialized agent analyzes failure traces from previous runs to suggest harness improvements.
• Harness Optimization Drills: Practice improving an agent's performance on standard benchmarks like Terminal Bench 2.0 or HumanEval solely through scaffolding and middleware changes
• MCP Integration Labs: Connecting an agent to a public MCP server (e.g., weather or GitHub) using MultiServerMCPClient

3. Where to Learn

• E-book: AI Agents and Applications (Chapter 11, Section 11.7–11.9 on multi-tool agents and step-by-step debugging, and Chapter 13 - Building and consuming MCP servers)
• Technical Case Studies: "Harness Engineering for Agentic Coding Systems" (ZenML/LangChain).
• Document: LangChain MCP
• SDK: FastMCP 2 (Python) for building custom MCP servers.

4. Subjective Outputs Required

• The Onboarded Developer Agent: A functional agent capable of mapping a complex directory and identifying its own tools via bash commands before attempting a task.
• The Loop-Detection Report: A documented instance where a LoopDetectionMiddleware successfully nudged an agent to reconsider its plan after a repeated failure.
• Verified Completion Trace: A LangSmith trace showing an agent attempting a task, failing an internal test, fixing its implementation, and successfully passing the PreCompletionChecklist.
• Harness Benchmark Report: Documented evidence of a significant jump in task accuracy (e.g., advancing from Pass@1 of 52% to 66%) achieved without swapping the underlying LLM, solely through harness engineering