Module 1.2: Engine Parameters (The "Knobs")

Curriculum Alignment: docs/plan/01_phase1_engine_and_prompting.md
Topic Scope: Temperature, Top-P (Nucleus Sampling), Stop Sequences, Max Length, Frequency & Presence Penalties
Level: Advanced AI Engineering / Architecture

1. How the Engine Generates Tokens: From Logits to Output

Before an LLM emits a token, the final linear transformer layer produces unnormalized scores called logits $z \in R^{| V |}$ for every token in vocabulary $V$ . How those logits are transformed into actual generated text is governed by sampling hyperparameters:

2. The Core Parameters (The "Knobs")

1. Temperature ( $T$ ): Controlling Creativity vs. Determinism

Temperature scales logits before the softmax activation function:

P (w_{i}) = \frac{\exp (z_{i} / T)}{\sum_{j \in V} \exp (z_{j} / T)}

$T = 0.0$ (Greedy Decoding / Argmax): $lim_{T \to 0} P (w_{i}) = {\begin{cases} 1 & if z_{i} = max (z) \\ 0 & otherwise \end{cases}$ The model deterministically selects the highest-probability token every single time.
- Best for: Factual Q&A, SQL queries, structured JSON extraction, and classification where consistency is critical.
$T = 0.7 - 0.8$ (Balanced Sampling): Retains the relative probability ranking while allowing minor variation.
- Best for: Creative writing, synthesis, conversational agents, brainstorming.
$T > 1.0$ (High Entropy): Flattens the probability curve, elevating the chances of low-probability tokens.
- Outcome: Rapidly increases hallucinations, syntactic drift, and incoherence.

2. Top- $P$ (Nucleus Sampling): Balancing Diversity

Top- $P$ dynamically truncates the probability distribution, discarding the long tail of low-probability tokens. Given tokens sorted by descending probability $p_{(1)} \geq p_{(2)} \geq \dots \geq p_{(| V |)}$ , Top- $P$ finds the smallest subset $V^{(p)}$ such that:

\sum_{i \in V^{(p)}} p_{(i)} \geq p

Top- $P = 0.9$ : The model considers only the top tokens that account for $90 %$ of the probability mass. The remaining $10 %$ tail is zeroed out and probabilities are renormalized.
Top- $P = 0.1$ : Only the top $10 %$ mass is considered (approximates greedy decoding).

IMPORTANT

Production Tuning Rule: Calibrate either Temperature or Top-P, but never tune both simultaneously in production pipelines. Altering both parameters simultaneously compounds variance across both the probability mass cutoff and distribution entropy.

3. Stop Sequences & Max Length

Stop Sequences: A list of strings (e.g. ["Observation:", "\n\n", "User:"]) that signal the engine to immediately halt generation.
- Critical Role: Stops runaway loops in agentic tool-calling workflows and prevents the model from generating fake conversational turns.
Max Length (max_tokens): The hard upper bound on the number of completion tokens generated in a single turn. Protects against infinite generation bugs and budget exhaustion.

4. Frequency & Presence Penalties

These hyperparameters penalize logits based on whether and how often tokens have already appeared in the current completion:

z_{i}^{'} = z_{i} - (α_{freq} \cdot c_{i}) - (α_{pres} \cdot 1_{{c_{i} > 0}})

Frequency Penalty ( $α_{freq}$ ): Scaled by the token's occurrence count $c_{i}$ . The more a specific word has appeared, the more heavily it is penalized.
- Best for: Eliminating repetitive loops (e.g. repeating the same phrase over and over in long outputs).
Presence Penalty ( $α_{pres}$ ): A flat penalty applied if a token has appeared at least once, regardless of frequency.
- Best for: Encouraging the model to introduce new topics and wider vocabulary in long-form generation.

3. Parameter Selection Matrix

Task Type	Recommended Temperature	Recommended Top-P	Stop Sequences	Penalties
Strict JSON Extraction	`0.0`	`1.0`	`["}"]` (optional)	`freq=0.0, pres=0.0`
Classification / Triage	`0.0`	`1.0`	`["\n"]`	`freq=0.0, pres=0.0`
Code Generation	`0.1 - 0.2`	`0.95`	Context-dependent	`freq=0.0, pres=0.0`
Conversational Agent	`0.7`	`1.0`	`["User:", "\n\n"]`	`freq=0.2, pres=0.1`
Creative Brainstorming	`0.9`	`1.0`	None	`freq=0.3, pres=0.3`

Conceptual Mindmap: Sampling Knobs

4. Python Simulation: Logit Manipulation & Sampling

python

"""
Logit Sampling Simulator
Demonstrates Softmax Temperature scaling, Top-P nucleus truncation, and categorical sampling.
"""
from typing import Sequence
import numpy as np
import numpy.typing as npt

def softmax(logits: npt.NDArray[np.float64], temperature: float) -> npt.NDArray[np.float64]:
    """Applies numerically stable softmax with temperature scaling."""
    if temperature <= 1e-5:
        probs = np.zeros_like(logits)
        probs[np.argmax(logits)] = 1.0
        return probs
    
    scaled = logits / temperature
    shifted = scaled - np.max(scaled)
    exp_vals = np.exp(shifted)
    return exp_vals / np.sum(exp_vals)

def apply_top_p(probs: npt.NDArray[np.float64], top_p: float) -> npt.NDArray[np.float64]:
    """Truncates probabilities to the nucleus mass defined by top_p."""
    sorted_idx = np.argsort(probs)[::-1]
    sorted_probs = probs[sorted_idx]
    
    cumulative_probs = np.cumsum(sorted_probs)
    cutoff_mask = cumulative_probs > top_p
    cutoff_mask[0] = False  # Keep at least the highest probability token
    
    truncated = sorted_probs.copy()
    truncated[cutoff_mask] = 0.0
    return truncated / np.sum(truncated)

def simulate_sampling(
    vocabulary: Sequence[str],
    logits: list[float],
    temperature: float,
    top_p: float = 1.0,
) -> dict[str, float]:
    arr_logits = np.array(logits, dtype=np.float64)
    probs = softmax(arr_logits, temperature)
    if top_p < 1.0:
        probs = apply_top_p(probs, top_p)
    return {token: round(float(prob), 4) for token, prob in zip(vocabulary, probs)}

if __name__ == "__main__":
    vocab = ["thought", "action", "observation", "answer", "fallback"]
    mock_logits = [5.2, 4.8, 2.1, 3.5, 1.0]

    print("--- Greedy Decoding (T = 0.0) ---")
    print(simulate_sampling(vocab, mock_logits, temperature=0.01))

    print("\n--- Balanced Creative (T = 0.7) ---")
    print(simulate_sampling(vocab, mock_logits, temperature=0.7))

    print("\n--- Nucleus Filtering (T = 1.0, Top-P = 0.8) ---")
    print(simulate_sampling(vocab, mock_logits, temperature=1.0, top_p=0.8))

5. Curated Reading & Canonical References

Resource	Canonical Reference & Link	Specific Focus Areas
Industry Standard Guide	Prompt Engineering Guide — Settings	Deep dive into `Temperature`, `Top_P`, and token sampling mechanics.
Interactive Playgrounds	OpenAI Playground & Hugging Face Flan-T5	Comparative parameter experimentation across closed (GPT) and open-source (Flan-T5) architectures.
Primary Curriculum Book	AI Agents and Applications (Google Drive)	Part 1, Chapter 2: Calibration of generation parameters for agent reliability.

6. Active Recall (Module 1.2 Flashcards)

Sampling KnobsClick or press Space to flip ↺

Why is it an anti-pattern to tune both Temperature and Top-P simultaneously in production?

Sampling Knobs • AnswerClick to flip back ↻

Altering both parameters simultaneously compounds variance across both the probability mass cutoff and distribution entropy, making deterministic debugging nearly impossible.

💡 Architect Takeaway: Fix Top-P=1.0 and calibrate Temperature, OR fix Temperature=1.0 and calibrate Top-P.

Sampling KnobsClick or press Space to flip ↺

What is the mathematical difference between Frequency Penalty and Presence Penalty?

z_i' = z_i - (α_freq · c_i) - (α_pres · 1_{c_i > 0})

Sampling Knobs • AnswerClick to flip back ↻

Frequency penalty scales linearly with the token's occurrence count c_i, penalizing repeated words more heavily the more they appear. Presence penalty applies a flat deduction once a token appears at least once, regardless of count.

💡 Architect Takeaway: Use Frequency Penalty to stop repetitive loop bugs; use Presence Penalty to encourage broader topic coverage.

7. Hands-on Engineering Exercises

Exercise 1.2: Parameter Stability Benchmark

Task: Submit the same logic-heavy problem (e.g., the "Strange Sequence": "Given sequence: 2, 3, 5, 9, 17, 33... What is the recursive rule, and what are the next three numbers?") to an LLM across 20 iterations at $T = 0.0$ and 20 iterations at $T = 1.0$ .
Verification Metric:
1. Percentage of identical responses at $T = 0.0$ (should be $\sim 100 %$ ).
2. Output variance, explanation diversity, and error rates at $T = 1.0$ .
Deliverable Link: Directly produces your Parameter Stability Report required in Phase 1.

Module 1.2: Engine Parameters (The "Knobs") ​

1. How the Engine Generates Tokens: From Logits to Output ​

2. The Core Parameters (The "Knobs") ​

1. Temperature (T): Controlling Creativity vs. Determinism ​

2. Top-P (Nucleus Sampling): Balancing Diversity ​

3. Stop Sequences & Max Length ​

4. Frequency & Presence Penalties ​

3. Parameter Selection Matrix ​

Conceptual Mindmap: Sampling Knobs ​

4. Python Simulation: Logit Manipulation & Sampling ​

5. Curated Reading & Canonical References ​

6. Active Recall (Module 1.2 Flashcards) ​