Embedding is the step where each text chunk is converted into a numerical vector. These vectors make semantic search possible — the retrieval system compares the meaning of a query to the meaning of each stored chunk by measuring vector distances.
The process is deterministic: the same input text always produces the same vector for a given model. In production, embedding is done in batches of 100–2000 chunks per API request to minimize overhead and total ingestion time.
Quality — Does it produce meaningful similarity scores for your domain? General models underperform on specialized content (medical, legal, code). Domain-fine-tuned models can be significantly better.
Dimensionality — Higher dimensions encode more nuance but require more storage and slightly slower search. 384–3072 dimensions are common.
Token limit — Most models accept 512–8192 tokens. Chunks exceeding the limit are silently truncated, losing content.
Cost and latency — API models (OpenAI, Cohere) have per-token costs and network latency. Local models (BGE, E5, SBERT) require compute but have no per-token cost.
Queries and documents must be embedded with the same model. Vectors from different models live in incompatible spaces — comparing them produces meaningless similarity scores. Switching models requires re-embedding the entire knowledge base. Plan for this cost before choosing.
The vector (for search), the original chunk text (for the LLM to read), and metadata (source, date, category). The LLM needs the actual words — not just the vector.
When a source document changes, its chunks must be re-embedded and their vectors replaced. Stale vectors from outdated content continue to be retrieved until explicitly deleted or overwritten.
Ask the AI assistant about embedding models, batch processing, the consistency rule, or how to handle document updates.