Module 1.4: Instrumentation & Tracing Hierarchy (LangFuse)

Curriculum Alignment: docs/plan/01_phase1_engine_and_prompting.md
Topic Scope: LangFuse Setup, The Run as the Atomic Unit, Reading Single-Turn Traces, Latency & Token Economics
Level: Advanced AI Engineering / Architecture

---

1. Why LLMs Require Semantic Tracing

When developing applications powered by LLMs, standard API monitoring (tracking only HTTP status codes and roundtrip server latency) is fundamentally inadequate:

• An LLM API invocation returns HTTP 200 OK even when the model outputs a severe hallucination, leaks system instructions, or fails schema parsing.
• Execution is non-deterministic: the exact same prompt can yield different outputs, token lengths, costs, and execution times across runs.

To inspect, debug, and govern AI systems, we implement Semantic Observability using tools like LangFuse:

1. Full Prompt Payloads: Capturing the exact rendered prompt (including system prompt and user variables) and the raw completion string.
2. Token Economics: Tracking prompt tokens (input) vs. completion tokens (output) and calculating per-query cost.
3. Execution Latency: Measuring Time-to-First-Token (TTFT) and Inter-Token Latency (ITL).

---

2. The Tracing Hierarchy: The Run as the Atomic Unit

In LangFuse, application execution is organized into a three-level hierarchy:

Hierarchy Breakdown:

• Trace: The top-level container representing an entire user turn, end-to-end request, or task execution.
• Span: A nested execution interval representing a discrete programmatic step (e.g. data preprocessing, vector search, regex validation, schema parsing).
• The Run / Generation (The Atomic Unit):
  • The actual invocation of the language model.
  • Captures the exact prompt string, raw response string, model identifier (gemini-2.5-flash), sampling hyperparameters (temperature, top_p), total latency, and exact token counts.

---

3. How to Read a Single-Turn Trace

When inspecting a single-turn trace in the LangFuse UI or via API, verify these four essential fields:

┌────────────────────────────────────────────────────────┐
│ SINGLE-TURN TRACE INSPECTOR                            │
├─────────────────┬──────────────────────────────────────┤
│ 1. INPUT        │ The complete rendered text sent to   │
│                 │ the LLM (System + Context + User).   │
├─────────────────┼──────────────────────────────────────┤
│ 2. OUTPUT       │ The exact raw string emitted by the  │
│                 │ model before downstream parsing.     │
├─────────────────┼──────────────────────────────────────┤
│ 3. LATENCY      │ Time-to-First-Token (TTFT) and total │
│                 │ generation duration (ms).            │
├─────────────────┼──────────────────────────────────────┤
│ 4. TOKEN COUNT  │ Prompt Tokens (In) vs                │
│                 │ Completion Tokens (Out).             │
└─────────────────┴──────────────────────────────────────┘

Profiling Metrics: TTFT vs. ITL

1. Time-to-First-Token (TTFT):
   • The time elapsed from sending the request until the first token streams back.
   • Reflects server queuing, prompt tokenization, and transformer prefill processing.
   • High TTFT indicates massive prompt token length or cold starts.
2. Inter-Token Latency (ITL):
   • The time required to generate each subsequent token during autoregressive decoding.
   • Reflects model parameter size and GPU memory bandwidth.
   • High ITL indicates high output verbosity or compute contention.

---

4. Production Python Instrumentation: LangFuse SDK

"""
LangFuse Instrumentation Example
Demonstrates setting up LangFuse, tracing runs, and scoring single-turn executions.
"""
import os
import time
from typing import Any
from langfuse import Langfuse
from langfuse.decorators import langfuse_context, observe
from pydantic import BaseModel

# Initialize LangFuse client
langfuse = Langfuse(
    public_key=os.environ.get("LANGFUSE_PUBLIC_KEY", "pk-dummy"),
    secret_key=os.environ.get("LANGFUSE_SECRET_KEY", "sk-dummy"),
    host=os.environ.get("LANGFUSE_HOST", "https://cloud.langfuse.com")
)

@observe(as_type="generation")
def call_model_generation(rendered_prompt: str, temperature: float = 0.0) -> str:
    """The atomic generation run: captures prompt, completion, and token metrics."""
    # Simulating LLM call
    mock_completion = '{"category": "PERFORMANCE_DEGRADATION", "urgency": 4}'
    
    # Update generation metadata directly in active context
    langfuse_context.update_current_observation(
        input=rendered_prompt,
        output=mock_completion,
        model="gemini-2.5-flash",
        model_parameters={"temperature": temperature},
        usage={"input": 180, "output": 22, "total": 202}
    )
    return mock_completion

@observe(name="single_turn_pipeline")
def run_single_turn_triage(user_query: str) -> dict[str, Any]:
    """Root trace orchestrating preprocessing, LLM generation, and evaluation scoring."""
    langfuse_context.update_current_trace(
        tags=["phase_1", "single_turn_demo"],
        metadata={"client": "web_portal"}
    )
    
    # 1. Format prompt
    prompt = f"System: Classify the support ticket.\nInput: <user_input>{user_query.strip()}</user_input>"
    
    # 2. Invoke model generation
    raw_output = call_model_generation(prompt, temperature=0.0)
    
    # 3. Attach evaluation score to the trace
    langfuse_context.score_current_trace(
        name="valid_json",
        value=1.0,
        comment="Successfully emitted valid JSON schema"
    )
    
    return {"status": "success", "output": raw_output}

if __name__ == "__main__":
    result = run_single_turn_triage("Our database connection pool is exhausted.")
    print("Execution Result:", result)
    langfuse.flush()

Conceptual Mindmap: Instrumentation & Tracing

---

5. Curated Reading & Canonical References

Resource	Canonical Reference & Link	Specific Focus Areas
Primary Observability Tool	LangFuse Tracing Documentation	Trace hierarchy, @observe() decorator mechanics, and generation token/cost attribution.
LangFuse Python SDK	LangFuse Python SDK Reference	Context managers, manual span manipulation, and score_current_trace().
OpenTelemetry Standard	OTel GenAI Semantic Conventions	Industry-standard span attributes (gen_ai.usage.prompt_tokens, gen_ai.system).

---

6. Active Recall (Module 1.4 Flashcards)

ObservabilityClick or press Space to flip ↺

What is the atomic unit of execution in LangFuse tracing and what does it record?

Question / Scenario

Observability • AnswerClick to flip back ↻

The Run (or Generation) is the atomic unit. It represents the actual LLM API invocation, recording the exact rendered prompt, raw model output, model name, hyperparameters (temperature, top_p), latency, and prompt vs completion token counts.

💡 Architect Takeaway: Every model inference must be captured as a distinct atomic Generation span.

Technical Explanation

Performance MetricsClick or press Space to flip ↺

What do Time-to-First-Token (TTFT) and Inter-Token Latency (ITL) indicate?

Question / Scenario

Performance Metrics • AnswerClick to flip back ↻

TTFT measures prompt processing, server queuing, and transformer prefill latency. ITL measures autoregressive generation speed per token, bounded by GPU memory bandwidth.

💡 Architect Takeaway: High TTFT indicates massive prompts; high ITL indicates large output length or compute bottleneck.

Technical Explanation

---

7. Hands-on Engineering Exercises

Exercise 1.4: Single-Turn Trace Inspection Drill

• Task: Implement a Python script using the LangFuse SDK that traces a single-turn classification call.
• Requirements:
  • Log prompt text, model response, and token counts.
  • Calculate and print the ratio of prompt tokens to completion tokens.
  • Attach a programmatic evaluation score (schema_validity=1.0) to the trace.