Every word an LLM produces comes from the same loop: predict the next token, append it, repeat. There is no planning stage, no outline, and no global sense of what the output will look like before generation begins.
The model generates one token at a time, left to right, until it produces a special end-of-sequence token or hits the max_tokens limit set by the application. Every essay, code block, poem, and JSON response comes from this single mechanism.
Receive the input sequence — everything so far: the system prompt, conversation history, user message, and any tokens already generated in this response
Compute a probability distribution — for every token in the model's vocabulary (50K–100K tokens), assign a probability: how likely is this token to come next?
Sample or select one token — using the chosen decoding strategy (greedy, sampling, top-k, top-p) pick one token from the distribution
Append the token — add the selected token to the sequence; it becomes part of the input for the next step
Repeat from Step 1 — the model now has one more token in its input and generates the next one
Stop — when the model generates an end-of-sequence token, or when the max_tokens limit is reached
Given the input "The capital of France is", the model might assign P("Paris") = 0.82, P("Lyon") = 0.04, P("the") = 0.03. With high-probability sampling, "Paris" is selected, appended, and the loop continues with "The capital of France is Paris" as the new input.
Trained on trillions of tokens of human text, the model has learned the statistical patterns of human writing with extraordinary precision. It knows that after "The capital of" a country name follows, that code blocks need closing braces, that conversations end with certain phrases.
Grammar, paragraph flow, code structure, and conversational tone — these all emerged from the training signal. Fluency is a consequence of next token prediction at scale, not a separately taught skill.
Truthfulness requires knowing what is true. Next token prediction optimizes for what token is statistically likely given the context — not for what is factually accurate. If a confident-sounding conclusion is statistically likely to follow the preceding text, the model will generate it — regardless of whether the evidence supports it. This is the root cause of hallucinations. The model is doing exactly what it was trained to do. Truth was never explicitly in the objective.
Always selects the single highest-probability token at each step.
Advantage: deterministic, fast, consistent — same input always produces same output.
Disadvantage: can produce repetitive, formulaic output. Picking the best next token at each step does not guarantee the best overall sequence.
Draws a token randomly according to the probability distribution — higher-probability tokens are more likely but not guaranteed.
Advantage: introduces variability, enabling more natural and creative output.
Disadvantage: non-deterministic — same input can produce different outputs on different runs.
Each token requires a full forward pass through all transformer layers. The model cannot generate token 10 until it has generated tokens 1–9. This is why LLM responses stream token by token — you see words appearing as they are generated. For long responses, this takes several seconds. In LLM applications, latency is dominated by generation length, not prompt processing. Shorter responses are faster; longer responses take more time, unavoidably.
Creative writing, brainstorming, generating multiple options for the user to choose from.
Conversational interactions where slightly different phrasing each time feels more natural.
Code generation that must be syntactically valid and correct.
Structured data extraction where the output format must be exact (JSON, CSV).
Use temperature = 0 or deterministic settings for these cases.
Ask the AI if you need help understanding next token prediction, the generation loop, why LLMs hallucinate, the difference between greedy decoding and sampling, or why responses appear to stream rather than arriving all at once.