Phase 1 Lab Guide: Building the Deliverables

Curriculum Source: docs/plan/01_phase1_engine_and_prompting.md
Target Code Location: labs/phase_1/
Evaluation Skill: lab-evaluator

---

🎯 Required Deliverables Specification

You will implement the following two deliverables from scratch when you are ready:

1. The Hardened Classifier (labs/phase_1/src/classifier.py):

   • A multi-class text categorization engine using Pydantic v2 domain schemas.
   • Hardened with NIST AML Unicode normalization, XML delimitation, canary token verification, and anti-extraction directives.
   • Must achieve $100\mathrm{\%}$ resistance against a standard 5-attack red-teaming suite while maintaining $>95\mathrm{\%}$ classification accuracy on benign inputs.

2. Parameter Stability Report (labs/phase_1/src/parameter_benchmark.py):

   • An automated testing script executing a logic puzzle across 20 iterations at $T=0.0$ and 20 iterations at $T=1.0$.
   • Generates a Markdown report (labs/phase_1/PARAMETER_STABILITY_REPORT.md) calculating agreement rates, lexical edit distance (Levenshtein), and token latency.

---

🏗️ Lab 1 Architecture: The Hardened Classifier

Acceptance Criteria:

• [ ] Uses Python 3.12+ type annotations and Pydantic v2 models (ConfigDict(frozen=True)).
• [ ] Catches prompt injection attempts without crashing.
• [ ] Never reveals the internal canary secret or system prompt.
• [ ] Fully verified by automated pytest test suite (labs/phase_1/tests/test_classifier.py).

---

🏗️ Lab 2 Architecture: Parameter Stability Benchmark

The Logic Benchmark Problems

To demonstrate how temperature ($T=0.0$ vs. $T=1.0$) affects deterministic multi-step reasoning, test one or both of the canonical logic problems referenced in the curriculum and textbook:

Option A: The Canonical "Strange Sequence" Problem (AI Agents and Applications, Ch. 2, p. 73)

Definition: A sequence of integers is defined as "strange" if and only if:

1. It contains at least two odd numbers ($\text{count}(\text{odd})\ge 2$), AND
2. The sum of all its odd numbers is divisible by 3 ($\sum \text{odd}(\mathrm{mod}3)=0$).

Test Input: 3, 4, 5, 7, 10, 18, 22, 24
Ground Truth Derivation:

• Odd numbers identified: [3, 5, 7] (Count = 3 $\ge 2$ ✅)
• Sum of odd numbers: $3+5+7=15$
• Divisibility: $15÷3=5$, remainder 0 ($15(\mathrm{mod}3)=0$ ✅)
• Expected Output: Strange (with complete reasoning steps)

Option B: The "Palindrome" Verification Problem

Definition: Determine whether an alphanumeric sentence is a strict palindrome, disregarding whitespace, punctuation, and letter casing. Test Input: "A man, a plan, a canal: Panama"
Expected Output: True (Normalized: "amanaplanacanalpanama")

Benchmark Metrics to Compute:

1. Semantic Agreement Rate: $\frac{\text{Number of correct answers}}{\text{Total runs (e.g. 20)}}\times 100\mathrm{\%}$ at $T=0.0$ vs. $T=1.0$.
2. Lexical Determinism (Variance): Percentage of responses strictly identical to the mode response.
3. Latency Profile: Mean latency and standard deviation (seconds) comparing TTFT and completion duration.

---

🚀 How to Run & Verify

# 1. Activate environment
source .venv/bin/activate

# 2. Run automated test suites
pytest labs/phase_1/tests/ -v

# 3. Run the benchmark generator
python labs/phase_1/src/parameter_benchmark.py

Once verified, you can use the notion-tracker skill to sync the results directly to your Notion page:

"Sync Phase 1 lab deliverables and parameter report to my Notion page."