Module 2.1: Logical Decomposition

Curriculum Alignment: docs/plan/02_phase2_engineering_of_reasoning.md
Topic Scope: Chain-of-Thought (CoT), KV-Cache Working Memory, Token Bloat Bottleneck, Chain of Draft (CoD)
Level: Advanced AI Engineering / Architecture

---

1. Why LLMs Need Working Memory: The Compute-per-Token Limit

In standard zero-shot prompting, we demand an immediate answer:

$$\text{Input: Complex Problem}⟶\text{Output: Final Answer}$$

Because an autoregressive transformer computes each output token using a fixed number of matrix multiplications per layer, it possesses a fixed compute budget per token. When asked to solve a multi-step logic problem directly, the model must compress all necessary intermediate deductions into the feed-forward layers of the single final answer token, causing catastrophic reasoning failure.

Chain-of-Thought (CoT) as External Compute Scaffolding

Chain-of-Thought guides models to solve problems by explicitly decomposing the problem into sequential steps:

$$\text{Input}⟶\text{Intermediate Step 1}⟶\text{Intermediate Step 2}⟶\cdots ⟶\text{Final Answer}$$

• Each emitted intermediate reasoning token is appended to the context window and cached in the GPU Key-Value (KV) cache.
• Subsequent self-attention layers compute cross-attention over these prior thoughts.
• The output stream acts as an external working memory register, providing the model with multiple additional forward passes of compute over the problem space.

---

2. The Token Bloat Problem & Chain of Draft (CoD)

While standard Chain-of-Thought ("Let's think step by step") dramatically improves reasoning accuracy, it introduces severe operational bottlenecks in production:

1. High Token Costs: Verbose explanations ("First, we must carefully observe that...") consume 500–1,500 reasoning tokens per query.
2. High Latency: In autoregressive generation, latency scales linearly with output token count:$$\text{Total Latency}=\text{TTFT}+(N_{\text{tokens}}\times \text{ITL})$$Generating 800 tokens of verbose reasoning at $30\text{ ms/token}$ adds $24\text{ seconds}$ of pure latency.

┌────────────────────────────────────────────────────────┐
│ CANONICAL TEST CASE: THE STRANGE SEQUENCE PROBLEM       │
│ (AI Agents and Applications, Chapter 2, Section 2.5.6) │
│ Input: "3, 4, 5, 7, 10, 18, 22, 24"                     │
├────────────────────────────────────────────────────────┤
│ STANDARD CHAIN-OF-THOUGHT (CoT)                        │
│ Tokens: ~85 | Latency: ~2.5s                           │
├────────────────────────────────────────────────────────┤
│ "The sequence has odd numbers: 3, 5, and 7.            │
│  Count of odd numbers is 3, which is at least two.     │
│  The sum of these odd numbers is 3 + 5 + 7 = 15.       │
│  Checking divisibility: 15 divided by 3 is 5 with zero │
│  remainder, so 15 is divisible by 3.                   │
│  Therefore, according to criteria, sequence is Strange"│
├────────────────────────────────────────────────────────┤
│ CHAIN OF DRAFT (CoD)                                   │
│ Tokens: ~18 (78% savings) | Latency: ~0.5s             │
├────────────────────────────────────────────────────────┤
│ "Draft:                                                │
│ - Odds: [3, 5, 7] (count=3 >= 2)                       │
│ - Sum: 3+5+7 = 15                                      │
│ - Div: 15 % 3 == 0                                     │
│ Final: Strange"                                        │
└────────────────────────────────────────────────────────┘

The Chain of Draft Solution

Chain of Draft (CoD) forces the model to emit a dense, sketch-like scratchpad containing only minimal mathematical steps, abbreviations, and symbolic equations before producing the final answer:

• Token Reduction: Slashes reasoning token volume by 60% to 80%.
• Latency Reduction: Drastically reduces roundtrip inference time ($N_{\text{tokens}}\times \text{ITL}$).
• Accuracy Parity: Matches the reasoning accuracy of standard verbose CoT across benchmark reasoning tasks.

Conceptual Mindmap: Logical Decomposition

---

3. Implementation Pattern: Prompting for Chain of Draft

"""
Chain of Draft (CoD) vs Chain of Thought (CoT) Implementation Patterns
"""

COT_PROMPT_TEMPLATE = """You are an expert analytical reasoning engine.
Solve the following multi-step logical deduction problem.
Think through the solution carefully step by step, explaining your complete reasoning before giving the final answer.

Problem: {problem}
"""

COD_PROMPT_TEMPLATE = """You are an expert analytical reasoning engine.
Solve the following multi-step logical deduction problem.

<reasoning_constraint>
Use Chain of Draft:
- Produce a minimalist draft of reasoning using abbreviations, equations, and concise bullet points only.
- Limit the draft strictly to under 5 bullet points and under 40 words total.
- Output the final result under [FINAL_ANSWER].
</reasoning_constraint>

Problem: {problem}
"""

---

4. Curated Reading & Canonical References

Resource	Canonical Reference & Link	Specific Focus Areas
Primary Curriculum Book	AI Agents and Applications (Google Drive)	Chapter 2, Section 2.5.6 (p. 73): Chain of Thought prompting, intermediate reasoning steps, and the Strange Sequence logic problem.
Foundational CoT Paper	Chain-of-Thought Prompting Elicits Reasoning (Wei et al., 2022)	Emergent reasoning abilities, token computation allocation, and multi-step inference.
Chain of Draft Paper	Chain of Draft: Thinking Fast with Less Tokens (arXiv:2502.18600)	Token reduction trade-offs ($\ge 50\mathrm{\%}$ latency savings) while matching CoT accuracy.
DAIR.AI Guide	Prompt Engineering Guide — Chain-of-Thought	Zero-shot CoT ("Let's think step by step") vs Few-shot CoT demonstrations.

---

5. Active Recall (Module 2.1 Flashcards)

Reasoning FoundationsClick or press Space to flip ↺

Why does standard Zero-Shot generation fail at complex multi-step reasoning while Chain-of-Thought succeeds?

Question / Scenario

Reasoning Foundations • AnswerClick to flip back ↻

Autoregressive transformers execute a fixed number of operations per token. In zero-shot prompting, the model must solve all reasoning steps within the single final token. CoT forces the model to emit intermediate tokens, effectively using its context window and KV-cache as an external working memory register.

💡 Architect Takeaway: Intermediate tokens provide additional computational steps for complex logic.

Technical Explanation

Token OptimizationClick or press Space to flip ↺

What is the core distinction between Chain-of-Thought (CoT) and Chain of Draft (CoD)?

CoD Token Volume ≈ 20-40% of standard CoT

Question / Scenario

Token Optimization • AnswerClick to flip back ↻

CoT generates verbose conversational prose explaining every step, creating high latency and token costs. CoD forces the model to emit a dense, sketch-like scratchpad (minimal equations, symbols, abbreviations) achieving equivalent reasoning accuracy while slashing token count by 60–80%.

💡 Architect Takeaway: Use CoD for low-latency, high-volume production reasoning pipelines.

Technical Explanation

---

6. Hands-on Engineering Exercises

Exercise 2.1: The Efficiency Benchmark Drill

• Goal: Build the foundation for your Efficiency Report deliverable.
• Task: Select a complex multi-step logical problem (e.g. logic grid deduction, multi-step resource balancing, or constraint satisfaction).
• Protocol:
  1. Execute the problem across 10 iterations using standard CoT and 10 iterations using CoD.
  2. Track prompt_tokens, completion_tokens, total latency (ms), and answer accuracy.
  3. Verify whether CoD achieves $\ge 50\mathrm{\%}$ token savings while maintaining equal correctness to CoT.