Tokens are the units of text that AI language models process. Rather than reading characters one at a time (too granular) or entire sentences at once (too coarse), models operate on chunks called tokens — which are roughly equivalent to words or word pieces.
Tokens are the fundamental currency of LLMs. Everything a model reads and everything it writes is measured in tokens. Understanding tokens helps you understand why models behave the way they do — including why they have usage limits, why certain inputs cost more than others, and why some text gets cut off.
Text: Hello, world!
Likely tokens: Hello / , / world / !
Token count: 4
The exact breakdown depends on the specific tokenizer used by the model — different models may split the same text differently.
AI models cannot process raw text the way humans read it. Computers work with numbers, not letters. The tokenization step converts text into a numerical representation that the model can actually compute with. Without this conversion, no language processing would be possible.
Tokens also set the boundaries of everything a model can do: how much text it can receive, how much it can produce, how much each request costs, and what falls inside or outside its active memory (the context window). Tokens are not just an implementation detail — they are the unit that governs the model's entire interaction with text.
Raw text — the user's prompt or document, as typed
Tokenization — the tokenizer splits the text into tokens based on the model's vocabulary
Token IDs — each token is replaced by its unique integer ID from the vocabulary table
Embeddings — each token ID is converted into a dense numerical vector that captures meaning and relationships
Model processing — the model processes these vectors through its layers and produces output vectors
Decoded output — output token IDs are converted back to text and returned to the user
The most common misconception about tokens is that they are the same as words. They are not. A single token can be less than a word, exactly a word, or a common short phrase. The tokenizer makes these decisions based on what appeared most frequently in its training data.
As a rough rule of thumb, 1 token is approximately 4 characters of English text, and 100 tokens is roughly 75 words. But this varies significantly across languages — languages with longer words, different scripts, or less training data often use more tokens to express the same content.
Common, short English words are usually a single token.
Long, rare, or technical words are split into pieces.
Seeing concrete examples is the best way to build intuition for how tokenization works. Notice how familiar, common words become single tokens, while compound words or proper nouns — even short ones — may be split.
Sentence: I love AI!
Tokens: I / love / AI / ! — 4 tokens
Sentence: ChatGPT helps students.
Tokens: Chat / GPT / helps / students / . — 5 tokens
Sentence: Backpropagation is complex.
Tokens: Back / prop / ag / ation / is / complex / . — 7 tokens
Key point: The same information expressed in simpler language often uses fewer tokens — which matters for both cost and context limits.
Modern LLMs use a strategy called subword tokenization — splitting text into pieces that are smaller than full words but larger than individual characters. The most common algorithm for this is Byte Pair Encoding (BPE), which analyzes training data to find the most frequently occurring sequences and makes those sequences tokens.
Subword tokenization solves a fundamental problem: if you only allow full words as tokens, any word not in the vocabulary is completely unrecognizable. By splitting unknown words into subword pieces that are in the vocabulary, the model can still process and generate unfamiliar words by combining known parts.
Problem: A new technology word like "microlearning" may not appear in the training vocabulary at all.
Solution: Split it into micro + learning — both are known tokens.
Benefit: The model can understand and generate the word without ever having seen it as a single unit in training.
More examples:
Punctuation marks are almost always treated as their own separate tokens. This is important because punctuation carries meaningful information — a period signals the end of a thought, a question mark changes the tone, and a comma affects rhythm and meaning. The model needs to see these as distinct units.
This also means that adding punctuation to a prompt increases its token count slightly — worth knowing when working with tight context windows or cost-sensitive applications.
Common punctuation tokens: . , ? ! : ; ( ) " '
Example: "Really?" → Really / ? — 2 tokens
Example: "Stop! Don't." → Stop / ! / Don / 't / . — 5 tokens
Note: Contractions like "don't" are often split at the apostrophe into Don and 't.
Once text is split into tokens, each token is looked up in a vocabulary table and replaced with an integer called a token ID. This is the numerical representation the model actually works with. The model never processes the word "cat" — it processes the number 2457 (or whatever ID that token has in its specific vocabulary).
These token IDs are then converted into dense numerical vectors called embeddings, which encode not just the identity of the token but also its relationships to other tokens. Words with similar meanings end up with similar embeddings, which is how the model understands that "king" and "queen" are related, or that "Paris" and "France" are related in a similar way to "Tokyo" and "Japan."
Text: cat
Token ID: 2457 (example — actual IDs vary by model)
Embedding: A vector of hundreds of floating-point numbers that encode the meaning and context of "cat" relative to all other words
Why this matters: The embedding is what allows the model to reason about word meaning, similarity, and relationships — not just the raw token string.
A model's vocabulary is its complete set of known tokens. Every token the model can process or generate must be in its vocabulary. Modern LLMs typically have vocabularies of 50,000 to 100,000+ tokens — large enough to cover most English words, common word parts, punctuation, numbers, and tokens from other languages.
The size of the vocabulary is a design choice with trade-offs. A larger vocabulary means more common words get their own tokens (reducing the number of tokens needed per sentence). A smaller vocabulary forces more splitting, but requires less memory to store the embedding table.
Tokenization is the full process of converting raw text into a sequence of token IDs ready for the model. It happens automatically before every model call — users never see it, but it happens every single time.
Different models use different tokenizers, even if they seem similar. This means that the same input text can produce different token counts, different splits, and different IDs depending on which model you are using. This is why token count estimates from one model do not always apply exactly to another.
Input text: AI is useful.
Step 1 — Split into tokens: AI / is / useful / .
Step 2 — Look up token IDs: [1000, 338, 5922, 13] (example IDs)
Step 3 — Convert to embeddings: Each ID becomes a vector of numbers
Step 4 — Feed to model: The model processes the sequence of vectors
The context window is the maximum number of tokens a model can consider at one time — both the input it receives and the output it generates. Think of it as the model's working memory: it can only see and reason about what fits within the window.
Context window sizes have grown dramatically over time. Early GPT models had context windows of around 2,000 tokens. Modern models support windows of 128,000 tokens or more — enough for several full-length novels. But even with large windows, there are practical limits: very long contexts can be slower and more expensive to process, and models sometimes "lose focus" on information from early in a very long context.
What happens when the context is exceeded: The oldest messages or content are dropped first — which is why a chatbot may seem to "forget" what was said earlier in a very long conversation.
Token limits create a practical constraint: you cannot simply paste an entire book, database, or large codebase into a prompt. Systems that need to work with large amounts of text use strategies to stay within limits while still accessing the most relevant content.
The most common technique for handling long documents is Retrieval-Augmented Generation (RAG): the long document is split into smaller chunks, stored in a search index, and only the most relevant chunks are retrieved and included in the prompt when a question is asked. This way, the model sees the right information without exceeding its context window.
Most commercial AI APIs charge based on token usage. Both input tokens (what you send to the model) and output tokens (what the model generates back) are counted and billed. This makes token awareness practically important for anyone building AI-powered applications.
The cost difference between input and output tokens matters: output tokens are typically more expensive than input tokens because generating each token requires a full forward pass through the model, while input tokens can be processed more efficiently in parallel. A response that is ten times longer than it needs to be is not just slower — it is significantly more expensive.
Everything you send to the model: your prompt, system instructions, conversation history, and any documents you include as context.
A long system prompt adds to input tokens on every request.
Everything the model generates: the full response text, including any reasoning steps shown.
Usually more expensive per token than input — and length is set by the model or a max_tokens limit you configure.
Every word in an LLM's response is generated one token at a time. The model decides when to stop by producing a special end-of-sequence token, or by hitting a max_tokens limit that you set. If you do not set a limit, the model will generate until it naturally finishes — which could be a few words or several pages.
Setting a max_tokens parameter in your API call is how you control output length programmatically. This is useful for keeping responses concise, controlling cost, and ensuring outputs fit within downstream systems that have their own size limits. A response cut off at max_tokens may end abruptly — it simply stops generating at that point, which can produce incomplete sentences.
Prompt: Explain AI in one sentence.
The model generates token by token:
AI → is → the → field → of → computer → science → focused → on → … → . → [END]
Total output tokens: Every generated piece, including spaces and punctuation, contributes to the output token count.
Practical tip: If you want a concise answer, say so in the prompt ("in one sentence", "in under 100 words") — the model tends to generate responses of the length it expects based on your framing.
Explain why a token is not the same as a word, and why this matters for understanding LLMs. Then describe what a context window is and what happens when text is too long to fit inside one.
A token is different from a word because... (think about how common vs. rare words are treated)
Subword tokenization solves the problem of... (unknown or rare words)
A context window is... (the model's working memory, measured in tokens)
When text exceeds the context window... (what gets dropped, and what strategies help)
Ask the AI if you need help understanding tokens, tokenization, subword tokens, punctuation tokens, token IDs, vocabulary, context windows, token limits, token cost, or output length.