Word embeddings are numerical representations of words. A word embedding turns a word into a vector, which is a list of numbers.
Word — cat
Embedding — [0.21, -0.44, 0.73, 0.10, ...]
Machine learning models cannot directly understand words. They need numbers. Word embeddings represent words as numbers while trying to preserve meaning.
Related Words — cat, kitten, dog — related to animals. A good embedding model places these close together in vector space.
Less Related — cat and airplane — a good embedding model places them farther apart.
Word embeddings can capture relationships between words based on how they are used in text.
• doctor and hospital
• student and school
• king and queen
• car and road
• teacher and classroom
Key Idea — Words that appear in similar contexts often receive similar embeddings.
A vector is a list of numbers. Each number is called a dimension.
Vector — [0.32, -0.18, 0.91, 0.07]
Small embedding — 50 dimensions
Larger embedding — 300 dimensions
Modern models — hundreds or thousands of dimensions
Purpose — Each dimension helps represent part of the word's usage pattern.
Embedding space is the mathematical space where word vectors are placed. Words with similar meanings are often closer together.
Close in embedding space — cat, kitten, dog, puppy
Farther apart — cat and refrigerator
Purpose — Embedding space helps models compare meaning using math.
Word embeddings allow models to compare how similar words are.
cat and kitten — High similarity
cat and airplane — Low similarity
Important — The exact similarity depends on the embedding model.
Cosine similarity is a common way to compare vectors. It measures whether two vectors point in a similar direction.
Words are more similar.
cat and kitten → high similarity
Words are less similar.
cat and table → lower similarity
Word embeddings are learned from context. Context means the words that appear around another word.
Sentence 1 — The cat slept on the sofa.
Sentence 2 — The kitten slept on the sofa.
Observation — cat and kitten appear in similar contexts.
Result — The model may learn similar vectors for them.
A common idea behind word embeddings is that words appearing in similar contexts tend to have similar meanings.
Sentence 1 — The doctor treated the patient.
Sentence 2 — The nurse helped the patient.
Observation — doctor and nurse appear in medical contexts.
Possible Result — Their embeddings may become related.
Bag of Words represents text using word counts. Word embeddings represent words using dense meaning-based vectors.
| Bag of Words | Word Embeddings |
|---|---|
| Uses word counts | Uses numerical vectors |
| Often sparse | Usually dense |
| Does not know cat and kitten are related | Can place cat and kitten close together |
| Simple and interpretable | Better at capturing meaning |
TF-IDF measures word importance in a document. Word embeddings represent word meaning.
Good for keyword importance.
May treat car and automobile as different features.
Good for semantic similarity.
May place car and automobile close together.
Word embeddings are usually dense vectors. A dense vector has many non-zero values.
[0.18, -0.42, 0.67, 0.91]
Word embeddings — most values are non-zero.
[0, 0, 1, 0, 0, 0, 1, 0]
Bag of Words and TF-IDF — often many zeros.
Dense vectors can capture more information in fewer dimensions than very large sparse vectors.
Bag of Words vocabulary — 20,000 words → 20,000 positions → most values: 0
Word embedding — 300 dimensions → most values: meaningful numbers
Benefit — More compact representation of meaning.
Word embeddings are learned from large text datasets by looking at how words appear near other words.
Sentence 1 — The dog barked loudly.
Sentence 2 — The puppy barked loudly.
Observation — dog and puppy appear in similar contexts.
Learning — The model adjusts vectors so related words become closer in embedding space.
A word embedding model can be trained by predicting words from context.
Sentence — The cat sat on the ___.
Possible Prediction — mat
Goal — Adjust word vectors so the model gets better at predicting words from context.
Word2Vec is a classic method for learning word embeddings from word context.
• CBOW — Predicts a word from surrounding words.
• Skip-gram — Predicts surrounding words from a word.
CBOW stands for Continuous Bag of Words. It predicts a target word using surrounding context words.
Sentence — The cat sat on the mat.
Context — The, sat, on
Target Word — cat
Goal — Use nearby words to predict the missing word.
Skip-gram does the opposite of CBOW. It uses one word to predict surrounding words.
Target Word — cat
Possible Surrounding Words — The, sat, on
Goal — Learn which words commonly appear near each other.
GloVe is another classic word embedding method. GloVe stands for Global Vectors.
Learns From — Word co-occurrence across a large text collection
Co-occurrence — How often words appear together
Example — doctor and hospital may appear together often → the model learns they are related.
A word embedding can have many dimensions, but the dimensions are not usually simple human labels.
cat — [0.12, -0.35, 0.88, 0.41, ...]
Important — Dimension 1 does not necessarily mean "animal." Dimension 2 does not necessarily mean "size."
Key Idea — The model learns useful numerical patterns.
Word embeddings can capture different kinds of relationships between words.
• Similar meaning
• Opposite meaning
• Same category
• Related topics
• Common context
• Functional relationship
Examples — king and queen, doctor and nurse, Paris and France, teacher and student
Word embeddings can sometimes capture analogy-like relationships.
king is to queen as man is to woman
Important — This does not mean the model understands like a human. It means the vectors learned useful text patterns.
Synonyms are words with similar meanings. Word embeddings can place synonyms close together.
• big and large
• small and tiny
• car and automobile
• happy and joyful
Why — These words often appear in similar contexts.
Words do not need to be synonyms to be related. Related words often appear in similar topics or situations.
• doctor and hospital
• teacher and classroom
• keyboard and computer
• coffee and mug
Key Idea — Embeddings can capture related context, not just exact synonym meaning.
Antonyms are words with opposite meanings. Embeddings may sometimes place antonyms close together because they appear in similar contexts.
Antonyms — hot and cold, good and bad, happy and sad
Why They May Be Close — "The drink is hot." / "The drink is cold." — The words appear in similar sentence positions, even though their meanings are opposite.
Note — This is one limitation of embeddings.
Polysemy means one word can have multiple meanings. Traditional word embeddings can struggle with this.
Word — bank
Meaning 1 — A financial institution
Meaning 2 — The side of a river
Problem — Static embeddings often give one vector per word, so one vector must represent multiple meanings.
Contextual embeddings represent a word differently depending on the sentence.
The bank approved the loan.
Meaning — Financial institution
The river bank was muddy.
Meaning — Side of a river
Contextual Embeddings — Can create different vectors for bank in each sentence.
| Static Embeddings | Contextual Embeddings |
|---|---|
| One vector per word | Vector changes based on sentence |
| Example: Word2Vec | Example: transformer models |
| Struggles with multiple meanings | Handles context better |
| Older NLP approach | Common in modern NLP |
Neural networks often use embedding layers. An embedding layer maps token IDs to vectors.
Token — cat
Token ID — 2457
Embedding Layer Output — [0.21, -0.44, 0.73, 0.10, ...]
Then — The model uses the vector for prediction.
An embedding layer is a trainable part of a neural network. It stores vectors for tokens and updates them during training.
Words — amazing, excellent, helpful
Possible Pattern — These words may become connected to positive sentiment during training.
Pretrained embeddings are learned from large datasets before being used in a specific task.
• Sentiment analysis
• Text classification
• Search
• Question answering
• Recommendation
Benefit — They can save training time and improve performance.
Fine-tuning means updating pretrained embeddings for a specific task or domain.
Word — cell
Biology context — A biological unit
Technology context — A spreadsheet cell or mobile network cell
Fine-Tuning — Helps embeddings adapt to the task domain.
Word embeddings can help sentiment models learn that related emotional words have similar meanings.
Positive Words — amazing, excellent, helpful, love
Negative Words — terrible, broken, confusing, hate
Example — great and excellent may have similar vectors.
Embeddings can help search systems find results by meaning, not just exact keywords.
Query — how to fix a broken laptop
Relevant Document — repairing a damaged computer
Embedding Connections — fix ↔ repairing, broken ↔ damaged, laptop ↔ computer
This is called — semantic search
Semantic similarity means similarity in meaning, even when the exact words are different.
Sentence 1 — How do I reset my password?
Sentence 2 — I forgot my login password.
Observation — The wording is different, but the meaning is related.
Embeddings — Can help detect this similarity.
Embeddings can be used as features for classification tasks.
• Spam or Not Spam
• Positive or Negative
• Topic category
• Support intent
• Toxic or Not Toxic
Benefit — The model can learn from meaning patterns, not just word counts.
Embeddings can help recommendation systems find related content.
Lesson — Introduction to neural networks
Related Lesson — Deep learning basics
Reason — Embeddings can show that the topics are related.
Use — Recommend similar lessons or articles.
Chatbots use embeddings to process user messages and generate useful responses.
User Message — Can you explain overfitting?
Modern Chatbots — Use contextual embeddings inside transformer models.
Translation models use embeddings to represent words or tokens in different languages.
English — cat
French — chat
Goal — Learn relationships between words across languages.
Benefit — Embeddings help map meaning from one language to another.
Embeddings depend on tokenization. A model first breaks text into tokens, then converts those tokens into embeddings.
Important — Different tokenizers can create different embeddings.
A vocabulary is the set of tokens a model knows. Each token in the vocabulary can have an embedding.
cat, dog, run, happy, un, ness
Each Token — Maps to a vector.
If a token is missing — The model needs a way to handle it.
Older word embedding methods may struggle with unknown words. Subword tokenization can help.
Unknown Word — microlearning
Possible Subwords — micro, learning
Benefit — The model can use embeddings for smaller known pieces instead of treating the whole word as unknown.
• Represent meaning better than word counts
• Capture similarity between words
• Help with synonyms
• Use dense vectors
• Work well with neural networks
• Support semantic search
• Improve many NLP tasks
• Can be pretrained and reused
• Can reduce sparse vector problems
• Can learn bias from training data
• Static embeddings struggle with multiple meanings
• Similar context does not always mean same meaning
• May not explain predictions clearly
• Need large text data to train well
• Can be affected by poor-quality data
• May not handle rare words well without subwords
• Meaning depends on training data and domain
Word embeddings learn from text data. If the text data contains biased patterns, embeddings can learn those patterns.
If certain jobs are unfairly associated with certain groups in training text, embeddings may reflect that pattern.
Risk — Bias in embeddings can affect downstream NLP systems.
Responsible AI — Requires testing and reducing harmful bias.
Word embeddings are useful mathematically, but they are not always easy for humans to interpret.
cat — [0.21, -0.44, 0.73, 0.10]
Problem — It is hard to say exactly what each number means.
Key Idea — Embeddings are powerful, but not always easy to explain.
Word embeddings represent individual words or tokens. Sentence embeddings represent full sentences or paragraphs.
| Word Embeddings | Sentence Embeddings |
|---|---|
| Represent one word or token | Represent a full sentence or paragraph |
| Example: cat | Example: The cat slept on the sofa. |
| Useful for word meaning | Useful for comparing larger text |
• Meaning matters
• Similar words should be related
• The model needs dense vectors
• You are using neural networks
• You want semantic search
• You want better features than word counts
• Synonyms matter
• The task is more complex than keyword matching
Examples — Sentiment analysis, search, chatbots, translation, and recommendations
• Full sentence meaning is needed
• Context changes word meaning
• Long documents must be compared
• Complex reasoning is required
• The task needs factual accuracy
• The system needs generated responses
Possible Better Choices — Sentence embeddings or transformer models
• Thinking embeddings are the same as word counts
• Assuming every embedding dimension has a simple meaning
• Forgetting that static embeddings give one vector per word
• Ignoring context
• Assuming similar vectors always mean same meaning
• Forgetting that embeddings can contain bias
• Using embeddings without checking task performance
• Comparing embeddings from different models as if they use the same space
• Ignoring tokenization
• Assuming embeddings understand like humans
• Word embeddings represent words as numerical vectors.
• A vector is a list of numbers.
• Word embeddings help models process meaning.
• Similar words often have similar embeddings.
• Embeddings are dense vectors, unlike sparse Bag of Words or TF-IDF vectors.
• Embeddings are learned from word context in large text datasets.
• Word2Vec and GloVe are classic word embedding methods.
• Static embeddings give one vector per word.
• Contextual embeddings change based on the sentence.
• Embeddings are useful for sentiment analysis, search, classification, chatbots, translation, and recommendations.
• Word embeddings can learn bias and may be hard to interpret.
Explain why word embeddings are better than Bag of Words for comparing the words "car" and "automobile."
Then explain one limitation of word embeddings, using the word "bank" as an example.
Ask the AI if you need help understanding word embeddings, vectors, embedding space, cosine similarity, Word2Vec, GloVe, static embeddings, contextual embeddings, semantic search, or how embeddings compare with Bag of Words and TF-IDF.