When a model samples from its full vocabulary at each step, very low-probability tokens can occasionally win the sample. These outlier tokens can derail an otherwise coherent response — a non-sequitur word, a mismatched phrase, a random character.
Top-k and top-p constrain the candidate pool before sampling. Instead of choosing from 50,000–100,000 tokens, the model only chooses from a filtered, higher-quality subset. This eliminates the long tail of nearly-impossible tokens from consideration.
Top-k sampling keeps only the k highest-probability tokens at each step and samples from those.
Compute probabilities — model assigns a probability to every token in its vocabulary
Keep the top k — all tokens outside the top k are set to probability zero; they cannot be selected
Renormalize — the remaining k tokens are rescaled so their probabilities sum to 1
Sample — one token is drawn from the filtered, renormalized distribution
Equivalent to greedy decoding. Always selects the single most probable token. Fully deterministic.
Allows moderate variety. The 50 most probable tokens are candidates. Unlikely tokens are excluded.
Top-k uses the same value regardless of how confident the model is. If P("Paris") = 0.92 and k = 50, the model still considers 49 nearly-impossible alternatives alongside a near-certain answer. Conversely, if the model is genuinely uncertain across 200 equally-likely tokens, k = 50 arbitrarily cuts off half of the legitimate options. Top-k does not adapt to the model's confidence.
Top-p sampling — also called nucleus sampling — solves the fixed-cutoff problem by adapting the candidate pool size to the model's actual confidence at each step.
Sort tokens by probability — from highest to lowest
Build the nucleus — add tokens from the top until their cumulative probability reaches p
Sample from the nucleus — only tokens inside the cumulative-probability threshold are eligible
P("Paris") = 0.92. With p = 0.9, the nucleus is just one token — very deterministic for this step.
10 tokens each have ~0.09 probability. With p = 0.9, roughly 10 tokens are needed to reach the threshold. More variety allowed.
Top-p automatically narrows when the model is confident and expands when the model is uncertain. It uses only as much diversity as the model's probability distribution actually warrants. This is why top-p tends to outperform top-k in practice — it is self-adjusting.
In most production deployments, all three parameters are set together. They apply in sequence:
Temperature is applied first — reshapes the logits before softmax, making the distribution sharper or flatter
Top-k is applied next — filters to only the k highest-probability tokens from the temperature-adjusted distribution
Top-p is applied after — further filters the remaining candidates based on cumulative probability
Sampling draws one token from what remains
General-purpose chat: temperature = 0.7, top_p = 0.9, top_k disabled
Code and structured output: temperature = 0.0–0.2, top_p = 1.0
Creative writing: temperature = 0.9–1.2, top_p = 0.95
Setting top_p = 1.0 disables nucleus filtering. Setting top_k = 1 or temperature = 0 gives greedy decoding.
Ask the AI if you need help understanding top-k sampling, top-p nucleus sampling, why top-p is generally preferred over top-k, how these parameters interact with temperature, or what settings to use for different types of tasks.