Bag of Words is a simple way to represent text for a machine learning model. It turns text into numbers by counting how often words appear.
Text → Split into words → Count the words → Create a numerical representation
Example Text: I love dogs. I love cats.
Word Counts: I = 2 · love = 2 · dogs = 1 · cats = 1
Machine learning models cannot directly understand raw text. They need numbers. Bag of Words turns words into numerical features.
Key Idea: Word counts can help a model recognize patterns in text.
A text representation is a way to turn text into a format a model can use. Bag of Words represents text using word counts.
Document 1: I love AI → Vector: 1, 1, 1, 0
Document 2: I love coding → Vector: 1, 1, 0, 1
Vocabulary: I · love · AI · coding
Each number shows whether the word appears or how often it appears.
A vocabulary is the list of unique words used by the Bag of Words model.
Document 1: cats are cute
Document 2: dogs are loyal
Vocabulary: cats · are · cute · dogs · loyal
Important: Each document is represented using this vocabulary.
A document vector is a list of numbers that represents a document. Each position in the vector matches a word in the vocabulary.
Vocabulary: cat · dog · fish
Document: cat dog cat
Vector: 2, 1, 0
cat appears 2 times · dog appears 1 time · fish appears 0 times
This example shows how three short documents can be converted into Bag of Words vectors.
Document 1: I love cats → Vector: 1, 1, 1, 0, 0
Document 2: I love dogs → Vector: 1, 1, 0, 1, 0
Document 3: Dogs love food → Vector: 0, 1, 0, 1, 1
Vocabulary: I · love · cats · dogs · food
Bag of Words usually counts how often each word appears.
Text: spam spam offer now
Vocabulary: spam · offer · now
Vector: 2, 1, 1
spam appears twice · offer appears once · now appears once
Binary Bag of Words only records whether a word appears or does not appear.
Text: spam spam offer now
Vocabulary: spam · offer · now
Count Vector: 2, 1, 1
Binary Vector: 1, 1, 1
Meaning: Each word appeared at least once.
Bag of Words can help detect spam emails by learning which words often appear in spam messages.
Spam-like Words: win · money · click · offer · free · now
Email: Win a free prize now → Possible Prediction: Spam
Email: Can we meet after class? → Possible Prediction: Not Spam
Bag of Words can help detect whether text sounds positive, negative, or neutral.
Positive Words: great · amazing · love · helpful · excellent
Negative Words: bad · terrible · hate · broken · confusing
Text: I love this app. It is amazing.
Possible Sentiment: Positive
Bag of Words can help classify documents by topic based on common words.
Sports Words: team · score · game · player · season
Technology Words: software · computer · AI · model · data
Health Words: doctor · patient · medicine · hospital · treatment
Goal: Use word patterns to predict the topic.
Search systems can use word matching to find documents related to a query.
Query: machine learning basics
Document 1: Machine learning models learn from data.
Document 2: Pizza recipes use cheese and dough.
Likely Better Match: Document 1 — it shares more words with the query.
The biggest limitation of Bag of Words is that it ignores word order.
Sentence 1: Dog bites man.
Sentence 2: Man bites dog.
Same Words: dog · bites · man
Problem: The meanings are different, but Bag of Words may represent them similarly.
Bag of Words does not deeply understand grammar or how words interact in a sentence.
Sentence: The student did not pass.
Important Meaning: not pass
Problem: A simple word-count model may not fully understand how "not" changes the meaning.
Bag of Words does not understand context well. The same word can have different meanings in different sentences.
Sentence 1: The bank approved my loan.
Sentence 2: The bank was covered in river water.
Problem: The word "bank" has different meanings, but Bag of Words sees the same word.
Bag of Words often creates sparse vectors. A sparse vector has many zeros.
Vocabulary Size: 10,000 words
Document: Uses only 50 unique words
Vector: Most positions are 0
Problem: Sparse vectors can be large and inefficient.
The vocabulary can become very large if the text is messy or not normalized.
Example Variations: hello · Hello · HELLO · hello! · helloo · hellooo
Problem: These may become separate vocabulary entries.
Result: The vocabulary grows too much, making models slower and harder to train.
Text preprocessing can reduce noise and make Bag of Words more useful.
Before: The cats were running quickly!
After: cat run quick
Lowercasing helps treat the same word consistently.
Before Lowercasing: AI · ai · Ai
After Lowercasing: ai · ai · ai
Benefit: Reduces duplicate word forms.
Removing punctuation can simplify word counts, but punctuation should not always be removed.
Before: hello · hello! · hello!!!
After Punctuation Removal: hello · hello · hello
Benefit: The words are counted together.
Warning: Punctuation can carry emotion or meaning.
Stopword removal can help focus on more meaningful words, but it should be used carefully.
Before: The article is about machine learning.
After: article machine learning
Risk: Removing a word like "not" can change meaning.
Example: "not good" → "good" — that is a serious problem for sentiment analysis.
Stemming helps Bag of Words treat related words as similar.
Words: running · runs · runner
Possible Stem: run
Benefit: Related word forms can be counted together.
Limitation: Stemming can create unnatural forms. Example: studies → studi
Lemmatization can group related words while keeping cleaner dictionary forms.
running → run
better → good
mice → mouse
Benefit: Cleaner than stemming in many cases. Useful when word meaning matters more.
Each row represents one document, and each column represents one word in the vocabulary.
| Document | cats | like | milk | dogs | bones | and | play |
|---|---|---|---|---|---|---|---|
| cats like milk | 1 | 1 | 1 | 0 | 0 | 0 | 0 |
| dogs like bones | 0 | 1 | 0 | 1 | 1 | 0 | 0 |
| cats and dogs play | 1 | 0 | 0 | 1 | 0 | 1 | 1 |
After text is converted into Bag of Words vectors, a machine learning model can use those vectors to learn patterns.
Example: If words like "free", "win", and "click" appear often, the model may predict spam.
Naive Bayes is often used with Bag of Words for simple text classification tasks.
Words Common in Spam: win · free · offer · click
Words Common in Normal Emails: meeting · class · schedule · project
Naive Bayes: Uses word frequencies to estimate the likely class.
In many machine learning libraries, Bag of Words is created using a tool often called CountVectorizer.
Example Documents: I love AI · I love coding
Output: Numerical vectors based on word counts
One-hot encoding often represents one word at a time. Bag of Words represents a whole document using word counts.
Vocabulary: cat · dog · fish
One-Hot for dog: 0, 1, 0
Bag of Words for "cat dog cat": 2, 1, 0
Bag of Words counts word frequency. TF-IDF also considers how important a word is across many documents.
Word: the
Bag of Words: May have a high count.
TF-IDF: Usually gives it lower importance because it appears everywhere.
Key Difference: TF-IDF adjusts for common words.
Baseline: A simple starting model used for comparison.
Bag of Words uses word counts. Word embeddings use dense vectors that can capture meaning and similarity.
Bag of Words: cat and kitten are different words.
Word Embeddings: cat and kitten may have similar vectors.
Main Difference: Embeddings can capture relationships better than simple word counts.
Bag of Words is still useful even though modern NLP often uses embeddings and transformers.
Example: Classifying support tickets by topic
Example: Understanding a complex chatbot conversation
Create a Bag of Words representation for these two documents:
Document 1: I love AI
Document 2: AI helps students
First create the vocabulary. Then create a word count vector for each document. Finally, explain one limitation of Bag of Words.
Ask the AI if you need help understanding Bag of Words, vocabulary, document vectors, word counts, sparse vectors, preprocessing, spam detection, topic classification, or the limitations of Bag of Words.