Phase 5 Lab Guide: Building the Deliverables

Curriculum Source: docs/plan/05_phase5_harness_engineering_mcp.md
Evaluation Skill: lab-evaluator

🎯 Required Deliverables Specification

When you are ready to write the implementation code for Phase 5, you will implement the following 4 deliverables:

Objective: Build an autonomous developer agent equipped with a LocalContextMiddleware that maps a target repository and verifies its own toolchains before executing a prompt.
Requirements:
- Automatically executes host environment discovery:
  - Verifies presence and versions of git, python3, and pytest.
  - Generates a compact top-level directory topology map.
  - Inspects git branch and uncommitted status.
- Injects the HostEnvironmentSpec into the system prompt context.
- Verified with an automated test confirming the agent accurately describes its host environment and uses the discovered Python binary path.

Objective: Implement and verify a LoopDetectionMiddleware that intercepts repetitive failing actions and forces the agent to break out of a "Doom Loop".
Requirements:
- Implements canonical action hashing: $SHA-256 (tool_name + canonical_args)$ .
- Maintains a sliding history buffer of recent tool executions.
- Intercepts and aborts execution when an identical action fails $\geq 2$ consecutive times.
- Injects an assertive system directive commanding the model to stop, re-evaluate observations, and synthesize a new approach.
- Delivers a documented report (loop_detection_report.md or automated test output) showing the intervention successfully preventing runaway token burn.

Objective: Construct a Build-Verify-Fix workflow that blocks premature task termination and verifies code changes via programmatic test runners.
Requirements:
- Connects the agent to an automated test execution gate running pytest.
- Simulates an initial code implementation that fails unit tests.
- Proves the harness intercepts the failure, injects the exact traceback into the model's context, and drives a successful code repair.
- Connects the PreCompletionChecklistMiddleware to verify all acceptance criteria are met before task exit.
- Captures a complete LangFuse trace showing the initial failure, the self-repair step, and the final green verification.

Objective: Conduct an empirical benchmark demonstrating that scaffolding and harness engineering elevate task accuracy without modifying model parameters.
Requirements:
- Prepares a curated benchmark suite of 20+ algorithmic or refactoring coding challenges (e.g. subset of HumanEval or repo-level bug fixes).
- Evaluates performance across two harness configurations:
  1. Baseline Raw Agent: Standard zero-shot prompting with raw bash/file tools and zero middleware.
  2. Engineered Harness Agent: LocalContext onboarding, Build-Verify-Fix closed loops, and LoopDetectionMiddleware.
- Produces a comprehensive benchmark report (harness_benchmark_report.md) demonstrating a quantifiable jump in Pass@1 accuracy (e.g. advancing from ~50% to 65%+) achieved purely through harness engineering.