Phase 1 Lab Guide: Building the Deliverables

Curriculum Source: [docs/plan/01_phase1_engine_and_prompting.md](file:///Users/huychau/Documents/working/training/ai/docs/plan/01_phase1_engine_and_prompting.md)
Target Code Location: labs/phase_1/
Evaluation Skill: [lab-evaluator](file:///Users/huychau/Documents/working/training/ai/.agents/skills/lab-evaluator/SKILL.md)

---

🎯 Required Deliverables Specification

You will implement the following two deliverables from scratch when you are ready:

1. The Hardened Classifier (labs/phase_1/src/classifier.py):

   • A multi-class text categorization engine using Pydantic v2 domain schemas.
   • Hardened with NIST AML Unicode normalization, XML delimitation, canary token verification, and anti-extraction directives.
   • Must achieve $100\mathrm{\%}$ resistance against a standard 5-attack red-teaming suite while maintaining $>95\mathrm{\%}$ classification accuracy on benign inputs.

2. Parameter Stability Report (labs/phase_1/src/parameter_benchmark.py):

   • An automated testing script executing a logic puzzle across 20 iterations at $T=0.0$ and 20 iterations at $T=1.0$.
   • Generates a Markdown report (labs/phase_1/PARAMETER_STABILITY_REPORT.md) calculating agreement rates, lexical edit distance (Levenshtein), and token latency.

---

🏗️ Lab 1 Architecture: The Hardened Classifier

Acceptance Criteria:

• [ ] Uses Python 3.12+ type annotations and Pydantic v2 models (ConfigDict(frozen=True)).
• [ ] Catches prompt injection attempts without crashing.
• [ ] Never reveals the internal canary secret or system prompt.
• [ ] Fully verified by automated pytest test suite (labs/phase_1/tests/test_classifier.py).

---

🏗️ Lab 2 Architecture: Parameter Stability Benchmark

The Logic Benchmark Problem

To clearly demonstrate the effect of temperature on deterministic reasoning, use a multi-step logic problem with an exact deterministic solution:

The "Strange Sequence" Problem:
"Given the word list: ['apple', 'banana', 'avocado', 'cherry', 'apricot', 'blueberry']:
1. Filter out all words that do not start with the letter 'a'.
2. Reverse the characters of each remaining word.
3. Sort the reversed words alphabetically.
4. Output the final list as a comma-separated string."

Benchmark Metrics to Compute:

1. Semantic Agreement Rate: $\frac{\text{Number of correct answers}}{\text{Total runs}}\times 100\mathrm{\%}$
2. Lexical Determinism (Variance): Percentage of responses identical to the mode response.
3. Latency Profile: Mean latency and standard deviation (seconds).

---

🚀 How to Run & Verify

# 1. Activate environment
source .venv/bin/activate

# 2. Run automated test suites
pytest labs/phase_1/tests/ -v

# 3. Run the benchmark generator
python labs/phase_1/src/parameter_benchmark.py

Once verified, you can use the notion-tracker skill to sync the results directly to your Notion page:

"Sync Phase 1 lab deliverables and parameter report to my Notion page."