FundamentalsBeginner

LLM Sampling Parameters: How Temperature, Top-P, and Top-K Control Creativity

Direct Answer & Overview

Temperature, Top-P (nucleus sampling), and Top-K are stochastic hyper-parameters that calibrate the probability distribution over next-token logits, allowing developers to balance deterministic precision (for code and JSON) against creative divergence.

1.Logits, Softmax, and Temperature Scaling

During inference, the final layer of a transformer produces raw logit scores for every token in its vocabulary (often 100,000+ words). A softmax function converts these logits into probabilities. The temperature parameter (T) divides the raw logits before softmax: P(i) = exp(z_i / T) / sum(exp(z_j / T)). When T approaches 0, the probability distribution sharpens drastically, forcing the model to pick only the single most likely token (argmax). When T > 1.0, the distribution flattens, increasing the likelihood of unconventional vocabulary choices.

2.Top-P (Nucleus Sampling) vs. Top-K Filtering

Top-K restricts candidate tokens to the K highest probability choices (e.g. K=40), but struggles when vocabulary context varies between narrow and wide distributions. Top-P (nucleus sampling) dynamically accumulates tokens in descending probability order until their cumulative sum reaches threshold P (e.g. P=0.90). If one token dominates with 95% confidence, only that token is considered; if uncertainty is high, a broader set of tokens is evaluated.

3.Production Parameter Presets for Different Use Cases

• Structured Data & SQL: temperature=0.0, top_p=1.0 (strict determinism, minimal syntax errors). • Customer Support FAQ: temperature=0.2, top_p=0.85 (grounded accuracy with natural phrasing). • Creative Copywriting & Brainstorming: temperature=0.8, top_p=0.95 (engaging variety without incoherent gibberish).

Configuring Temperature and Top-P for Deterministic Code Generationpython

from openai import OpenAI

client = OpenAI(
    base_url="https://api.apihundred.com/v1",
    api_key="your_api100_key"
)

# For code generation, use temperature=0 for zero randomness
code_response = client.chat.completions.create(
    model="claude-3-5-sonnet",
    messages=[
        {"role": "user", "content": "Write a regex that validates ISO-8601 UTC timestamps."}
    ],
    temperature=0.0,
    top_p=1.0
)
print("Deterministic Output:", code_response.choices[0].message.content)

Frequently Asked Questions

Should I alter both temperature and top_p simultaneously?

It is standard engineering practice to tune either temperature or top_p, leaving the other at default (e.g. adjust temperature with top_p=1.0, or set temperature=1.0 and adjust top_p).

Does setting temperature=0 guarantee 100% identical outputs?

In theory yes, but GPU non-deterministic floating-point math across parallel threads can occasionally produce slight variations unless seed pinning and dedicated deterministic engines are enforced.

What happens if temperature is set above 1.5?

High temperatures flatten the probability curve so heavily that the model begins sampling nonsensical, out-of-context tokens and grammatical gibberish.

A100

API100 Engineering Team

Infrastructure & Latency Research

Author Profile →

Previous Guide

Video Understanding & Generation APIs: Sora, Gemini Video, and Temporal Transformers

Next Guide

System Prompts vs User Prompts: Steering AI Behavior, Personas, and Guardrails