Module 2.1: Logical Decomposition — Chain-of-Thought (CoT) vs. Chain of Draft (CoD)

Curriculum Alignment: [docs/plan/02_phase2_engineering_of_reasoning.md](file:///Users/huychau/Documents/working/training/ai/docs/plan/02_phase2_engineering_of_reasoning.md)
Topic Scope: Step-by-Step Reasoning, Transformer Autoregressive Planning, Token Bloat vs. Chain of Draft, Latency Economics
Level: Senior Architect / Advanced AI Engineering

---

1. Architectural Mental Model: Why LLMs Need Working Memory

In traditional computing, an algorithm uses RAM to store intermediate variables before returning a result. In standard zero-shot prompting, however, we ask an LLM:

$$\text{Input: Complex Problem}⟶\text{Output: Final Answer}$$

Because an autoregressive transformer computes each output token using a fixed number of matrix multiplications per layer, it has a fixed compute budget per token. If you ask an LLM to solve a 5-step logic puzzle directly, it must compute all 5 steps inside the feed-forward layers of the single final token. This causes catastrophic reasoning failure.

Chain-of-Thought (CoT) as External Compute Scaffolding

By forcing the model to generate intermediate tokens:

$$\text{Input}⟶\text{Intermediate Step 1}⟶\text{Intermediate Step 2}⟶\text{Final Answer}$$

Each emitted token is appended to the context window and passed into the Key-Value (KV) cache. The model literally uses its output stream as an external working memory register, providing additional compute passes over the problem.

---

2. The Token Bloat Problem & Chain of Draft (CoD)

While standard Chain-of-Thought ("Let's think step by step") dramatically improves reasoning accuracy, it introduces severe architectural drawbacks in production:

1. High Token Costs: Verbose explanations ("First, we need to carefully examine...") consume $500\text{--}1,500$ reasoning tokens per request.
2. High Latency: In autoregressive generation, latency scales linearly with output token count:$$\text{Total Latency}=\text{TTFT}+(N_{\text{tokens}}\times \text{ITL})$$Generating 800 tokens of verbose reasoning at $30\text{ ms/token}$ adds $24\text{ seconds}$ of pure latency.

The Chain of Draft (CoD) Solution

Chain of Draft is a standard technique developed in 2025/2026 to achieve the accuracy of CoT while slashing token volume by 60% to 80%.

Instead of verbose conversational explanations, CoD forces the model to emit a dense, sketch-like scratchpad containing only minimal mathematical steps, symbols, or single-word variables before outputting the final answer.

┌────────────────────────────────────────────────────────┐
│ STANDARD CHAIN-OF-THOUGHT (CoT)                        │
│ Tokens: ~240 | Latency: ~7.2s                          │
├────────────────────────────────────────────────────────┤
│ "First, let's look at the orders table. We need to sum │
│ all transactions for user 492. Next, we check the      │
│ discounts table to see if any promo codes apply. We    │
│ notice promo SUMMER20 is active, which gives 20% off.  │
│ 100 minus 20% is 80. Therefore, the final total is $80"│
└────────────────────────────────────────────────────────┘
                          VS
┌────────────────────────────────────────────────────────┐
│ CHAIN OF DRAFT (CoD)                                   │
│ Tokens: ~42 | Latency: ~1.2s                           │
├────────────────────────────────────────────────────────┤
│ "Draft:                                                │
│ - u492 orders: $100                                    │
│ - promo SUMMER20: -20%                                 │
│ - 100 * 0.8 = 80                                       │
│ Final: $80"                                            │
└────────────────────────────────────────────────────────┘

---

3. Implementation Pattern: Prompting for Chain of Draft

"""
Chain of Draft (CoD) vs Chain of Thought (CoT) Prompt Templates
"""

COT_PROMPT_TEMPLATE = """You are a senior software architect.
Solve the following microservices capacity planning problem.
Think through the solution carefully step by step, explaining your complete reasoning before giving the final numbers.

Problem: {problem}
"""

COD_PROMPT_TEMPLATE = """You are a senior software architect.
Solve the following microservices capacity planning problem.

<reasoning_constraint>
Use Chain of Draft:
- Produce a minimalist draft of calculations using abbreviations and symbols only.
- Limit the draft to strictly under 5 bullet points and under 40 words total.
- Output the final result under [FINAL_SPEC].
</reasoning_constraint>

Problem: {problem}
"""

---

4. Curated Reading & Canonical References

Resource	Canonical Reference & Link	Specific Focus Areas
Primary Curriculum Book	AI Agents and Applications (Google Drive)	Chapter 2, Section 2.5 ("Reasoning in Detail"): Cognitive architecture foundations and step-by-step decomposition.
Foundational CoT Paper	Chain-of-Thought Prompting Elicits Reasoning (Wei et al., 2022)	Emergent reasoning abilities, token computation allocation, and multi-step inference.
Chain of Draft Paper	Chain of Draft: Thinking Fast with Less Tokens (arXiv:2502.18600)	Token reduction trade-offs ($\ge 50\mathrm{\%}$ latency savings) while matching CoT accuracy.
Advanced Guides	Patronus AI: Advanced Prompt Engineering	Structural reasoning prompting, constraint enforcement, and drift reduction.
DAIR.AI Guide	Prompt Engineering Guide — Chain-of-Thought	Zero-shot CoT ("Let's think step by step") vs Few-shot CoT demonstrations.

---

5. Senior Exercises

Exercise 2.1: The Efficiency Benchmark Drill

• Goal: Prepare your methodology for the required Efficiency Report.
• Task: Select a multi-step capacity planning problem (e.g. calculating RPS, database connection pool exhaustion, and network throughput under failover).
• Protocol:
  1. Formulate 10 distinct variations.
  2. Test CoT vs. CoD.
  3. Track prompt_tokens, completion_tokens, and answer correctness.
  4. Verify if CoD achieves $\ge 50\mathrm{\%}$ token savings without degrading calculation accuracy.