LLM Sampling Parameters: How Temperature, Top-P, and Top-K Control Creativity

Overview #

Temperature, Top-P (nucleus sampling), and Top-K are stochastic hyper-parameters that calibrate the probability distribution over next-token logits, allowing developers to balance deterministic precision (for code and JSON) against creative divergence.

Logits, Softmax, and Temperature Scaling #

During inference, the final layer of a transformer produces raw logit scores for every token in its vocabulary (often 100,000+ words). A softmax function converts these logits into probabilities. The temperature parameter (T) divides the raw logits before softmax: P(i) = exp(z_i / T) / sum(exp(z_j / T)). When T approaches 0, the probability distribution sharpens drastically, forcing the model to pick only the single most likely token (argmax). When T > 1.0, the distribution flattens, increasing the likelihood of unconventional vocabulary choices.

Top-P (Nucleus Sampling) vs. Top-K Filtering #

Top-K restricts candidate tokens to the K highest probability choices (e.g. K=40), but struggles when vocabulary context varies between narrow and wide distributions. Top-P (nucleus sampling) dynamically accumulates tokens in descending probability order until their cumulative sum reaches threshold P (e.g. P=0.90). If one token dominates with 95% confidence, only that token is considered; if uncertainty is high, a broader set of tokens is evaluated.

Production Parameter Presets for Different Use Cases #

• Structured Data & SQL: temperature=0.0, top_p=1.0 (strict determinism, minimal syntax errors).
• Customer Support FAQ: temperature=0.2, top_p=0.85 (grounded accuracy with natural phrasing).
• Creative Copywriting & Brainstorming: temperature=0.8, top_p=0.95 (engaging variety without incoherent gibberish).

Code Example: Configuring Temperature and Top-P for Deterministic Code Generation #

from openai import OpenAI

client = OpenAI(
    base_url="https://api.apihundred.com/v1",
    api_key="your_api100_key"
)

# For code generation, use temperature=0 for zero randomness
code_response = client.chat.completions.create(
    model="claude-3-5-sonnet",
    messages=[
        {"role": "user", "content": "Write a regex that validates ISO-8601 UTC timestamps."}
    ],
    temperature=0.0,
    top_p=1.0
)
print("Deterministic Output:", code_response.choices[0].message.content)

Frequently Asked Questions #

Q: Should I alter both temperature and top_p simultaneously?

It is standard engineering practice to tune either temperature or top_p, leaving the other at default (e.g. adjust temperature with top_p=1.0, or set temperature=1.0 and adjust top_p).

Q: Does setting temperature=0 guarantee 100% identical outputs?

In theory yes, but GPU non-deterministic floating-point math across parallel threads can occasionally produce slight variations unless seed pinning and dedicated deterministic engines are enforced.

Q: What happens if temperature is set above 1.5?

High temperatures flatten the probability curve so heavily that the model begins sampling nonsensical, out-of-context tokens and grammatical gibberish.