Tokenization is the process of breaking text into smaller pieces called tokens. A token can be a word, part of a word, punctuation mark, number, symbol, or even a character.
Text: Machine learning is powerful.
Tokens: Machine · learning · is · powerful · .
Computers do not understand full sentences the way humans do. Before a model can process text, the text usually needs to be broken into smaller parts and converted into numbers.
A token is one piece of text after tokenization. Different tokenizers may split the same text in different ways.
Text: I love NLP!
Possible Tokens: I · love · NLP · !
The exclamation mark can be its own token.
Word tokenization breaks text into words. It is common in traditional NLP methods such as Bag of Words and TF-IDF.
Text: Natural language processing is useful.
Word Tokens: Natural · language · processing · is · useful
Common Use: Traditional NLP models and search systems
In word tokenization, each word usually becomes a separate token.
Text: The student asked a question.
Tokens: The · student · asked · a · question · .
The period may also become a token depending on the tokenizer.
Sentence tokenization breaks a paragraph into separate sentences.
Paragraph: NLP is useful. It helps computers work with language.
Sentence 1: NLP is useful.
Sentence 2: It helps computers work with language.
Use: Processing long text one sentence at a time
Sentence tokenization is useful when a model or system needs to analyze one sentence at a time.
Example: A document assistant may split a long article before finding the most relevant sentence.
Subword tokenization breaks words into smaller word pieces. It is common in modern language models.
Word: unhappiness
Possible Subword Tokens: un · happy · ness
Key Idea: A model can understand parts of a word, even if the full word is rare.
Subword tokenization helps models handle rare words, new words, and word variations.
Word: unbelievable
Possible Subwords: un · believ · able
Why Useful: Even if "unbelievable" is rare, the model may know the smaller pieces.
Character tokenization breaks text into individual characters.
Text: cat
Character Tokens: c · a · t
Strength: Can handle any word.
Limitation: Creates longer sequences, which can be slower to process.
Punctuation can become its own token. This can be useful because punctuation may carry meaning.
Text: Hello, world!
Tokens: Hello · , · world · !
? = Shows a question. ! = Shows emotion. , = Affects sentence structure.
Numbers can also become tokens. Whether numbers matter depends on the task.
Text: I scored 95 on the exam.
Tokens: I · scored · 95 · on · the · exam · .
Number matters: "I give this 5 stars." — the rating is important.
Number may not matter: "User 847392 opened the file." — the ID is noise.
Whitespace tokenization splits text wherever there is a space. It is simple, but it has limitations.
Text: I love machine learning
Tokens: I · love · machine · learning
Problem — Text: Hello, world!
Whitespace Tokens: Hello, · world!
Issue: The comma and exclamation mark stay attached.
Rule-based tokenization uses rules to split text. These rules may handle spaces, punctuation, contractions, numbers, symbols, and abbreviations.
Text: I can't go.
Possible Tokens 1: I · can't · go · .
Possible Tokens 2: I · ca · n't · go · .
Important: Different rules create different token outputs.
Contractions can be tricky because they combine multiple meanings into one written form.
Text: I don't know.
Possible Tokenization: I · do · n't · know · .
Why Useful: The token "n't" helps preserve negation, which changes the meaning.
Hyphenated words can be kept together or split into smaller tokens.
Text: state-of-the-art model
Tokens: state-of-the-art · model
Treats the whole phrase as one unit.
Text: state-of-the-art model
Tokens: state · - · of · - · the · - · art · model
Exposes the individual words inside.
Emojis can become tokens. They may be useful because they often carry emotion.
Text: I love this app 😊
Tokens: I · love · this · app · 😊
Sentiment Clue: The emoji suggests positive emotion.
URLs can be kept as tokens, split, removed, or replaced with a special token.
Text: Visit https://example.com now!
Tokens: Visit · https://example.com · now · !
Good for: Spam detection where URLs are signals.
Text: Visit https://example.com now!
Tokens: Visit · <URL> · now · !
Good for: Topic classification where URLs are noise.
Hashtags are common in social media text. Splitting them can reveal meaning inside the hashtag.
Text: I love #MachineLearning
Tokens: I · love · #MachineLearning
Treats the full hashtag as one token.
Text: I love #MachineLearning
Tokens: I · love · # · Machine · Learning
Reveals the words inside the hashtag.
Mentions are common in social media and chat platforms. A system may keep, remove, or replace them.
Text: Thanks @Alex for the help.
Possible Tokens: Thanks · @Alex · for · the · help · .
Alternative: Thanks · <USER> · for · the · help · .
Traditional NLP often uses word tokenization before creating features such as word counts or TF-IDF values.
Modern NLP models often use subword tokenization. The tokenizer is usually built to match the model.
Why Useful: They help models handle rare words, new words, different languages, and word variations.
A vocabulary is the set of tokens a model knows.
cat · dog · run · happy · un · ness · .
Process: Text is tokenized. Each token is matched to the vocabulary.
An unknown token represents a word or token the model does not recognize.
Unknown Token: <UNK>
Known Words: cat · dog · run
Unknown Word: zyxtable → <UNK>
Subword tokenization reduces this problem by splitting rare words into smaller known pieces.
Models usually convert tokens into token IDs. A token ID is a number that represents a token.
cat → Token ID: 2457
dog → Token ID: 3891
The exact token IDs depend on the tokenizer used by the model.
Text: I love cats.
Tokens: I · love · cats · .
Example Token IDs: 45 · 928 · 3412 · 13
Important: Different models may use different IDs for the same text.
Padding is used when sequences need to be the same length in a batch.
I · love · NLP
After Padding:
I · love · NLP · <PAD> · <PAD>
Machine · learning · is · very · useful
No Padding Needed.
Padding helps models process batches efficiently.
Truncation cuts text when it is too long for a model's token limit.
Maximum Token Length: 10 tokens
Text Length: 20 tokens
Truncation: Only part of the text is kept.
Risk: Important information may be removed.
Many models have token limits. A token limit is the maximum number of tokens the model can process at once.
Model limit: 512 tokens
Document length: 2,000 tokens
Possible Solutions:
Token limits matter for: chatbots, summarization, document search, and question answering.
Tokenization affects how much context can fit into a model.
Long Document: May become many tokens.
If Model Has a Token Limit: Only part of the document can fit.
Common Solution: Split long documents into chunks.
Goal: Keep useful context while staying within the limit.
Chunking means splitting long text into smaller sections that fit within a token limit.
Search systems often tokenize both the query and the documents.
Query: machine learning basics
Query Tokens: machine · learning · basics
Document Tokens: machine · learning · models · learn · from · data
Goal: Compare tokens to find relevant documents.
Sentiment analysis depends on words, punctuation, and sometimes emojis.
Text: This app is not good!!!
Tokens: This · app · is · not · good · ! · ! · !
Important Tokens: not · good · !
Risk: Removing or splitting tokens incorrectly can change the meaning.
Chatbots tokenize user messages before generating responses.
User Message: Can you explain neural networks?
Tokens: Can · you · explain · neural · networks · ?
Model: Processes input tokens and generates output tokens.
Final Step: Output tokens are converted back into text.
Translation systems tokenize the source language and generate tokens in the target language.
Input: Good morning
Input Tokens: Good · morning
Output Token: Bonjour
Goal: Preserve meaning across languages.
Some NLP models work with code. Code tokenization must preserve symbols because small symbols can change meaning.
Code: x = 5
Tokens: x · = · 5
Code Tokenization May Include: keywords · variable names · operators · symbols · strings · numbers
Good tokenization keeps useful meaning while making text easier for the model to process.
Important: There is no single best tokenizer for every task.
Tokenization is one part of text preprocessing. Text preprocessing includes many steps, while tokenization specifically means splitting text into tokens.
This example keeps punctuation, negation, and emoji because they may be useful for sentiment analysis.
Text: "Wow!!! I can't believe this AI tool works so well 😊"
Possible Tokens: Wow · ! · ! · ! · I · ca · n't · believe · this · AI · tool · works · so · well · 😊
Useful Meaning Kept:
Misconception: Tokens are always words.
Reality: Tokens can be words, subwords, punctuation marks, numbers, symbols, or characters.
Misconception: Tokenization is the same for every model.
Reality: Different models use different tokenizers.
Misconception: Removing punctuation before tokenization is always good.
Reality: Punctuation can carry meaning.
Misconception: Longer text is always better.
Reality: Models have token limits, and long text may need chunking.
Tokenize the sentence below in two different ways: first using word tokenization, then using a tokenizer that keeps punctuation as separate tokens.
"I can't believe this chatbot works so well!"
Then explain why keeping the exclamation mark and the contraction information might matter for sentiment analysis.
Ask the AI if you need help understanding tokenization, word tokens, subword tokens, sentence tokens, character tokens, token IDs, padding, truncation, chunking, or how tokenization affects NLP models.