Sentence embeddings encode an entire sentence or paragraph into a single fixed-length vector. The key advance over word embeddings is context-awareness: the representation of each word is influenced by all the other words in the sentence.
"I went to the bank to deposit money" and "I sat on the bank of the river" produce different sentence embeddings that correctly capture their different meanings — even though they share the word "bank."
Sentence embedding models use transformer architectures. Transformers process all words simultaneously using attention mechanisms — each word attends to all other words and builds a contextual representation. The per-token representations are then pooled (typically by averaging) into a single fixed-length vector.
SBERT (Sentence-BERT, 2019) is the most influential sentence embedding model. Its innovation was the training objective: a siamese network trained with contrastive loss to pull similar sentence pairs together and push dissimilar pairs apart in vector space.
Required one forward pass per sentence pair. Comparing n sentences required n×(n-1)/2 passes. Impractical for large-scale retrieval.
Requires one forward pass per sentence. Embeddings are precomputed and compared with a dot product. Scales to millions of documents.
Local models (all-MiniLM, all-mpnet) — Run on CPU, no API cost, full control. Best for privacy-sensitive data or budget constraints.
API models (OpenAI text-embedding-3-small/large) — No setup, high quality, per-token cost. Best for quick deployment.
Domain-specific models — Fine-tuned on medical, legal, or scientific text. Outperform general models for specialized domains.
Always embed queries and documents with the same model. Vectors from different models live in incompatible spaces — comparing them produces meaningless results.
Ask the AI assistant about sentence embeddings, SBERT, or how to choose an embedding model.