Word embeddings assign a fixed vector to each word in a vocabulary, where the position in vector space encodes relationships to other words. The classic models — Word2Vec, GloVe, and FastText — learn these representations by training on large text corpora.
"You shall know a word by the company it keeps." Words that appear in similar contexts — surrounded by similar other words — get similar vectors. This is how word embeddings encode meaning without any explicit definitions.
Word2Vec trains a shallow neural network on a prediction task. In Skip-gram, the model sees a word and predicts the surrounding context words. In CBOW, it sees context words and predicts the center word. The prediction task is a pretext — the real output is the learned word vectors, which capture semantic relationships as a side effect of training.
king − man + woman ≈ queen
Paris − France + Italy ≈ Rome
These relationships emerge from patterns in text without any explicit supervision about word relationships. The vector difference encodes a relational concept that transfers across domains.
Each word gets exactly one vector, regardless of context. "Bank" in "deposit money at the bank" and "bank of the river" receive identical vectors — even though they mean completely different things. This is why modern systems use contextual embeddings from transformer models instead.
Word2Vec — Neural network trained on local context prediction. Fast and effective. Context window is local.
GloVe — Learns from global word co-occurrence statistics across the whole corpus. Combines local and global information.
FastText — Represents words as bags of character n-grams. Can generate vectors for unseen words by composing sub-word components. Best for morphologically rich languages.
Ask the AI assistant about word embeddings, Word2Vec, GloVe, or why contextual embeddings supersede them.