Temperature

At a temperature approaching zero, sampling collapses towards always choosing the highest-probability token, which is as close to deterministic as a model call gets — though batching and floating-point non-determinism mean it is not a guarantee. As temperature rises, tokens further down the distribution become reachable, which is what makes output feel creative and also what lets a wrong-but-plausible continuation get selected. Top-p and top-k truncate the distribution before sampling and interact with temperature, so tuning all three at once tends to produce confusion rather than insight.

The mapping to real tasks is direct. Extraction, classification, structured output, tool argument generation and anything a downstream parser depends on want low temperature. Brainstorming, alternative phrasings and generating diverse candidates for a later ranking step want higher values. If a pipeline needs both, run the divergent step hot and the converging step cold rather than compromising on one setting for the whole chain.

The misconception is that temperature controls accuracy or confidence. It does neither. It changes how much of the model’s existing distribution you are willing to sample from — a model that is confidently wrong at high temperature is confidently wrong at low temperature too, just more consistently. Lowering temperature makes errors reproducible, which is genuinely useful for debugging, but it does not make them less likely to be errors.