Temperature and Sampling

Temperature is a parameter used when a language model converts its output probabilities into actual tokens. Before sampling, the model's raw logit scores are divided by the temperature value T; a lower temperature sharpens the distribution toward the most likely tokens, while a higher temperature flattens it so that less likely tokens gain probability mass.

How It Works

At each generation step a large language model (LLM) produces a score, or logit, for every token in its vocabulary. These logits are turned into a probability distribution with the softmax function. Temperature inserts an extra term into that function: each logit is divided by T before the softmax is applied.^[1]

At T = 1 the distribution is unchanged from what the model was trained to produce. As T approaches 0, the distribution collapses toward its single highest-probability token — the output becomes deterministic, so repeated runs of the same prompt return the same text. As T rises above 1, the distribution flattens and improbable tokens become selectable, producing more varied but also less coherent output.^[2]

Temperature does not change what the model knows; it changes how boldly the model samples from what it knows. A model asked a factual question usually performs best at low temperature, while brainstorming, creative writing and code-variation tasks typically use 0.7 to 1.2.

Sampling in Practice

Temperature is rarely used alone. Most inference stacks combine it with a truncation strategy that first removes clearly unsuitable candidates:

• Top-k sampling restricts the draw to the k most likely tokens, then samples from that shortened list.^[3]
• Top-p (nucleus) sampling keeps the smallest set of tokens whose cumulative probability exceeds a threshold p, such as 0.9.^[4]
• Greedy decoding is the special case of choosing the argmax token every step, equivalent to T = 0 without randomness.

Deterministic decoding methods such as beam search search for the highest-probability sequence rather than sampling, and remain common in translation and summarisation, though sampling with tuned temperature has become the default for chat models.

Temperature also interacts with repetition penalties and logit bias settings exposed by inference APIs. OpenAI, Anthropic and Google publish recommended ranges for their models, and platforms such as OpenRouter, Together AI and vLLM expose temperature as a standard request field.^[5]

Why It Matters

Because temperature is cheap to change at runtime — it requires no retraining — it is the primary lever engineers use to shift a single deployed model between formats: a factual retrieval answer at T = 0.2, a conversational assistant at T = 0.7, and an ideation tool at T = 1.1. Evaluation teams therefore fix the temperature when benchmarking, since a variable sampling setting makes scores non-reproducible. Poorly tuned temperature is also a source of production defects: too high and models hallucinate or drift from instructions, too low and outputs become repetitive loops.^[6]

>Key Takeaways

• Temperature rescales token logits before sampling; low values make output deterministic, high values make it more diverse.
• It is normally combined with top-k or top-p truncation to cut off very unlikely tokens.
• It changes no model weights, so it can be tuned per request at zero training cost.
• Fix temperature for benchmarks, and keep it low where outputs must be reproducible for audit.

See Also

• Nucleus Sampling
• Token
• Large Language Models
• Inference
• Prompt Engineering
• Hallucination