Module 1.1: The Non-Deterministic Engine & Parameter Knobs

Curriculum Alignment: [docs/plan/01_phase1_engine_and_prompting.md](file:///Users/huychau/Documents/working/training/ai/docs/plan/01_phase1_engine_and_prompting.md)
Topic Scope: Autoregressive Next-Token Mechanics, Spiky Intelligence, Sampling Knobs ( $T$ , Top- $P$ , Penalties)
Level: Senior Architect / Advanced AI Engineering

1. Architectural Mental Model: LLMs as Stochastic Microservices

In traditional distributed systems, microservices are deterministic: for a given idempotent request $R$ , the service returns response $S$ subject only to network or database latency.

An Autoregressive Large Language Model (LLM), however, is a stochastic next-token predictor:

P (x_{t} ∣ x_{< t}; θ)

Where $θ$ represents the frozen billions of parameters, and the output is a continuous probability distribution over a discrete vocabulary $V$ (typically $\sim 32, 000$ to $128, 000$ token IDs).

The "Spiky Intelligence" Phenomenon

As software architects, we expect systems to exhibit continuous capability curves (e.g., if a system can solve complex graph optimization, it can surely perform elementary string reversing). LLMs break this intuition due to Byte-Pair Encoding (BPE) tokenization and non-symbolic weight representation:

Why Tokenization Causes Spikes: An LLM does not see characters or bytes; it sees token IDs. The word "strawberry" might be tokenized as ["str", "aw", "berry"]. Asking a frontier model how many 'r's are in "strawberry" requires it to reconstruct character boundaries from abstract vector embeddings, leading to elementary failures despite solving graduate-level distributed consensus problems.
Architecture Takeaway: Never rely on an LLM for operations that a deterministic compiler, regex, or Python standard library function can perform in $O (1)$ time. Treat the model as a probabilistic semantic reasoning engine, wrapped by deterministic harness middleware.

2. Mathematical Mechanics of the Engine Knobs

Before an LLM emits a token, the final linear layer produces a raw vector of unnormalized scores called logits $z \in R^{| V |}$ . How those logits turn into actual generated tokens is controlled by sampling algorithms and hyperparameters:

1. Temperature ( $T$ )

Temperature controls the "entropy" or spread of the probability distribution:

P (w_{i}) = \frac{\exp (z_{i} / T)}{\sum_{j \in V} \exp (z_{j} / T)}

$T \to 0$ (Greedy Decoding / Argmax): $lim_{T \to 0} P (w_{i}) = {\begin{cases} 1 & if z_{i} = max (z) \\ 0 & otherwise \end{cases}$ The model deterministically selects the highest-probability token. Use case: Structured JSON extraction, code compilation, database query generation, classification.
$T = 1.0$ (Neutral Softmax): The probability distribution reflects the true pre-trained distribution.
$T > 1.0$ (High Entropy / "Creative"): The logit differences are flattened, raising the probability of low-ranked tokens. Leads to hallucinations, diversity, and linguistic drift.

2. Top- $P$ (Nucleus Sampling)

Instead of considering all $| V |$ tokens, Top- $P$ dynamically cuts off the long tail of low-probability tokens. Given sorted probabilities $p_{(1)} \geq p_{(2)} \geq \dots \geq p_{(| V |)}$ , find the smallest subset $V^{(p)}$ such that:

\sum_{i \in V^{(p)}} p_{(i)} \geq p

Tokens outside $V^{(p)}$ have their probabilities zeroed out, and the remaining tokens are renormalized.

Top- $P = 0.9$ : Restricts the model to the top $90 %$ probability mass. If the top 2 tokens already account for $91 %$ mass, only those 2 tokens are considered.
Top- $P = 0.1$ : Extremely narrow selection (almost greedy).

IMPORTANT

Architectural Rule: Change either Temperature or Top-P, but rarely both simultaneously in production pipelines. Altering both simultaneously introduces uncalibrated sampling variance.

3. Presence and Frequency Penalties

These modify logits at runtime based on the historical token counts $c_{i}$ in the current generation:

z_{i}^{'} = z_{i} - (α_{freq} \cdot c_{i}) - (α_{pres} \cdot 1_{{c_{i} > 0}})

Frequency Penalty ( $α_{freq}$ ): Scaled by token frequency count. Discourages repeating specific words/phrases.
Presence Penalty ( $α_{pres}$ ): Flat penalty applied if a token has appeared at least once. Encourages topical exploration.

3. Python Simulation: Logit Manipulation & Sampling

The following script simulates the exact mathematical pipeline inside an LLM's decoding loop:

python

"""
Logit Sampling Simulator
Simulates Softmax Temperature scaling, Top-P nucleus truncation, and categorical sampling.
"""
from typing import Sequence
import numpy as np
import numpy.typing as npt

def softmax(logits: npt.NDArray[np.float64], temperature: float) -> npt.NDArray[np.float64]:
    """Applies numerically stable softmax with temperature scaling."""
    if temperature <= 1e-5:
        # Emulate greedy argmax
        probs = np.zeros_like(logits)
        probs[np.argmax(logits)] = 1.0
        return probs
    
    scaled_logits = logits / temperature
    # Subtract max for numerical stability (avoids overflow in exp)
    shifted = scaled_logits - np.max(scaled_logits)
    exp_logits = np.exp(shifted)
    return exp_logits / np.sum(exp_logits)

def apply_top_p(probs: npt.NDArray[np.float64], top_p: float) -> npt.NDArray[np.float64]:
    """Truncates probabilities to the nucleus mass defined by top_p."""
    sorted_indices = np.argsort(probs)[::-1]
    sorted_probs = probs[sorted_indices]
    
    cumulative_probs = np.cumsum(sorted_probs)
    # Identify indices to remove (tokens beyond the threshold)
    cutoff_mask = cumulative_probs > top_p
    # Keep at least the highest probability token
    cutoff_mask[0] = False
    
    truncated_probs = sorted_probs.copy()
    truncated_probs[cutoff_mask] = 0.0
    
    # Renormalize
    return truncated_probs / np.sum(truncated_probs)

def simulate_sampling(
    vocabulary: Sequence[str],
    logits: list[float],
    temperature: float,
    top_p: float = 1.0,
) -> dict[str, float]:
    arr_logits = np.array(logits, dtype=np.float64)
    probs = softmax(arr_logits, temperature)
    if top_p < 1.0:
        probs = apply_top_p(probs, top_p)
    return {token: round(float(prob), 4) for token, prob in zip(vocabulary, probs)}

# Execution Example
if __name__ == "__main__":
    vocab = ["microservice", "monolith", "serverless", "event-driven", "batch"]
    mock_logits = [5.2, 4.8, 2.1, 3.5, 1.0]

    print("--- Greedy (T = 0.01) ---")
    print(simulate_sampling(vocab, mock_logits, temperature=0.01))

    print("\n--- Balanced (T = 0.7) ---")
    print(simulate_sampling(vocab, mock_logits, temperature=0.7))

    print("\n--- High Entropy (T = 1.8) ---")
    print(simulate_sampling(vocab, mock_logits, temperature=1.8))

    print("\n--- Nucleus Filtering (T = 1.0, Top-P = 0.75) ---")
    print(simulate_sampling(vocab, mock_logits, temperature=1.0, top_p=0.75))

4. Curated Reading & Canonical References

Review these primary sources specified in the curriculum plan:

Resource	Canonical Reference & Link	Specific Focus Areas
Primary Curriculum Book	AI Agents and Applications (Google Drive)	Part 1, Chapters 1–2: Foundations of LLM engine behavior, stochastic generation, and agent primitives.
Foundational Course	DeepLearning.AI: Generative AI for Everyone	Intuitive mental models for autoregressive token mechanics and enterprise capability boundaries.
Industry Standard Guide	Prompt Engineering Guide — Settings	Deep dive into `Temperature`, `Top_P`, and token sampling mechanics.
Interactive Playgrounds	OpenAI Playground & Hugging Face Flan-T5	Comparative parameter experimentation across closed (GPT) and open-source (Flan-T5) architectures.
Government Security Standard	NIST AI 100-2e2025: Adversarial Machine Learning	Section 2.1: Taxonomy of evasion, poisoning, and stochastic variance vulnerabilities.

5. Hands-on Engineering Exercises

Exercise 1.1: Temperature Variance Profiler

Task: Write a Python script using pytest that submits the same multi-step logic problem (e.g., "Reverse the words in: 'microservices domain driven design' and count the total vowels") to an LLM across 20 iterations at $T = 0.0$ and 20 iterations at $T = 1.0$ .
Verification Metric: Calculate:
1. Percentage of identical responses at $T = 0.0$ (should be $\sim 100 %$ ).
2. Entropy and output variance at $T = 1.0$ .
Deliverable Link: This builds the foundation for your Parameter Stability Report required in Phase 1.

Module 1.1: The Non-Deterministic Engine & Parameter Knobs ​

1. Architectural Mental Model: LLMs as Stochastic Microservices ​

The "Spiky Intelligence" Phenomenon ​

2. Mathematical Mechanics of the Engine Knobs ​

1. Temperature (T) ​

2. Top-P (Nucleus Sampling) ​

3. Presence and Frequency Penalties ​

3. Python Simulation: Logit Manipulation & Sampling ​

4. Curated Reading & Canonical References ​

5. Hands-on Engineering Exercises ​

Exercise 1.1: Temperature Variance Profiler ​