Appearance
5.3 Aggressive Self-Verification & Build-Verify-Fix Loops
Canonical Curriculum Reference:
docs/plan/05_phase5_harness_engineering_mcp.md
Textbook Reference: AI Agents and Applications (Manning Publications), Chapter 11, Section 11.6 ("Step-by-step debugging", pp. 308–310) & Section 11.8 (pp. 312–315).
🚫 1. The Pathology of "Vibe-Based" Agent Verification
When autonomous agents are instructed to write code or modify architectures, they suffer from a well-documented cognitive pathology: The Hallucination of Competence.
When an LLM completes an edit, its autoregressive weights naturally transition to generating affirming language:
"I have successfully implemented the feature. All unit tests and boundary conditions are handled cleanly!"
In reality, without programmatic execution:
- Subtle syntax errors, unimported modules, and misspelled variable names go undetected.
- Edge cases and off-by-one errors break runtime behavior.
- The model declares "completion" based purely on linguistic plausibility rather than actual empirical verification.
The Engineering Solution: The Closed-Loop Build-Verify-Fix Cycle
To achieve production-grade reliability, the harness must strip the model of the authority to declare completion based on its own opinion. Instead, completion can only be certified by deterministic external tools (compilers, type checkers, and test runners).
+--------------------------------------------+
| 1. SPECIFICATION |
| (User Feature Request & Acceptance Tests) |
+--------------------------------------------+
|
v
+--------------------------+
| 2. CODE GENERATION / |
| MODIFICATION |
+--------------------------+
|
v
+--------------------------+
| 3. PROGRAMMATIC TEST |
| EXECUTION (Pytest) |
+--------------------------+
|
+---------------+---------------+
| |
[Tests Fail] [Tests Pass]
| |
v v
+-----------------------+ +-----------------------+
| 4. TRACEBACK FEEDBACK| | 5. VERIFIED RELEASE |
| (Inject raw failure | | (Artifact Signed & |
| into LLM context) | | Committed) |
+-----------------------+ +-----------------------+
|
+-----> (Loop to Step 2, Max N Attempts)⚙️ 2. Architectural Mechanics: The Programmatic Test Gate
The Build-Verify-Fix loop requires four non-negotiable operational invariants:
- Isolated Test Execution: Tests run in an isolated subprocess with explicit timeouts (e.g. 15 seconds) to prevent infinite loops in generated code.
- Lossless Error Capture: Raw standard error, standard out, and exit codes are captured without summarization. Compressing or paraphrasing tracebacks robs the LLM of exact line numbers and exception signatures.
- Targeted Repair Prompting: The fix prompt must instruct the model to diagnose the root cause rather than rewriting entire unrelated modules.
- Bounded Iteration Guard: An absolute recursion budget (e.g.
) to prevent runaway token expenditure when a specification contains contradictory requirements.
💻 3. Production Implementation: BuildVerifyFixLoop
Below is an enterprise Python 3.12 implementation utilizing Pydantic v2 domain models and defensive subprocess orchestration:
python
import subprocess
from pathlib import Path
from typing import Final
from pydantic import BaseModel, ConfigDict, Field
class TestExecutionResult(BaseModel):
"""Immutable result of a deterministic test suite execution."""
model_config = ConfigDict(frozen=True, extra="forbid")
passed: bool = Field(description="True if test runner exited with code 0")
exit_code: int = Field(description="Process return code")
stdout: str = Field(description="Captured standard output")
stderr: str = Field(description="Captured standard error")
duration_seconds: float = Field(description="Execution wall-clock time")
class VerificationCycleReport(BaseModel):
"""Audit report of a complete Build-Verify-Fix cycle."""
model_config = ConfigDict(frozen=True, extra="forbid")
attempts_used: int
success: bool
final_output: str
error_history: list[str] = Field(default_factory=list)
class BuildVerifyFixOrchestrator:
"""Enforces aggressive self-verification via programmatic test runners."""
def __init__(
self,
workspace_root: Path,
max_repair_attempts: int = 3,
timeout_seconds: int = 15
) -> None:
self._workspace: Final[Path] = workspace_root.resolve()
self._max_attempts: Final[int] = max_repair_attempts
self._timeout: Final[int] = timeout_seconds
def execute_test_gate(self, test_target: str) -> TestExecutionResult:
"""Executes pytest against the specified file or directory."""
import time
start_time = time.monotonic()
cmd = ["pytest", test_target, "-v", "--tb=short"]
try:
res = subprocess.run(
cmd,
cwd=self._workspace,
capture_output=True,
text=True,
timeout=self._timeout,
check=False
)
duration = time.monotonic() - start_time
return TestExecutionResult(
passed=(res.returncode == 0),
exit_code=res.returncode,
stdout=res.stdout,
stderr=res.stderr,
duration_seconds=round(duration, 3)
)
except subprocess.TimeoutExpired:
return TestExecutionResult(
passed=False,
exit_code=-1,
stdout="",
stderr=f"Test runner timed out after {self._timeout} seconds (Possible infinite loop).",
duration_seconds=float(self._timeout)
)
def run_cycle(
self,
test_file: str,
code_editor_callback, # Callable[[str | None], None]
) -> VerificationCycleReport:
"""Runs the iterative build-verify-fix loop."""
error_history: list[str] = []
for attempt in range(1, self._max_attempts + 1):
# 1. First iteration generates code; subsequent iterations receive traceback
last_error = error_history[-1] if error_history else None
code_editor_callback(last_error)
# 2. Deterministic Verification Gate
test_res = self.execute_test_gate(test_file)
if test_res.passed:
return VerificationCycleReport(
attempts_used=attempt,
success=True,
final_output=f"Verification certified on attempt {attempt} via pytest.",
error_history=error_history
)
# 3. Capture failing traceback for feedback
failure_diagnostic = (
f"[Attempt {attempt} FAILED with exit code {test_res.exit_code}]\n"
f"STDOUT:\n{test_res.stdout}\n"
f"STDERR:\n{test_res.stderr}"
)
error_history.append(failure_diagnostic)
return VerificationCycleReport(
attempts_used=self._max_attempts,
success=False,
final_output=f"Verification failed after exhausting {self._max_attempts} attempts.",
error_history=error_history
)🧪 4. Automated Testing & Verification
python
import pytest
from pathlib import Path
from tempfile import TemporaryDirectory
def test_build_verify_fix_success_flow():
with TemporaryDirectory() as tmp_dir:
root = Path(tmp_dir)
test_file = root / "test_sample.py"
src_file = root / "calc.py"
# Test expecting add(2, 3) == 5
test_file.write_text("from calc import add\ndef test_add(): assert add(2, 3) == 5")
attempt_counter = 0
def mock_llm_editor(error_feedback: str | None):
nonlocal attempt_counter
attempt_counter += 1
if attempt_counter == 1:
# First attempt: Buggy code
src_file.write_text("def add(a, b): return a * b")
else:
# Second attempt: Repaired code
assert error_feedback is not None
assert "assert add(2, 3) == 5" in error_feedback
src_file.write_text("def add(a, b): return a + b")
orchestrator = BuildVerifyFixOrchestrator(workspace_root=root, max_repair_attempts=3)
report = orchestrator.run_cycle(str(test_file), mock_llm_editor)
assert report.success is True
assert report.attempts_used == 2
assert len(report.error_history) == 1📈 Impact on Benchmark Accuracy
Empirical results from SWE-bench and HumanEval benchmarks demonstrate that introducing a deterministic programmatic verification gate elevates single-turn agent accuracy significantly:
| Verification Architecture | Pass@1 Accuracy | Average Attempts | Hallucinated Completions |
|---|---|---|---|
| Zero-Verification (Raw Output) | |||
| Self-Reflection (LLM critic) | |||
| Programmatic Build-Verify-Fix |