Retrieval finds the chunks most relevant to the user's query. If retrieval fails, the LLM cannot generate a correct answer — regardless of how powerful it is. Retrieval quality is the primary bottleneck in most RAG systems.
Embed the query → search the vector database for the k most similar chunk vectors → return the top-k chunks to the context injection step. The query is embedded with the same model used during document ingestion.
k is how many chunks are retrieved per query. Too small and a single bad retrieval derails the answer. Too large and weak results dilute the good ones or exceed the LLM's context window.
Typical production values: k = 3 to 10. Calibrate on real queries.
Similarity threshold — Only return chunks exceeding a minimum cosine similarity score. When nothing meets the threshold, return "I don't have information on this" instead of hallucinating from weak retrievals.
Metadata filtering — Scope retrieval to specific subsets: by product, date, category, or access level. Applied during ANN search for efficiency.
Query rewriting — Use the LLM to expand or clarify a short or ambiguous query before embedding. "deployment issue" → "common causes of deployment failures in Kubernetes."
Re-ranking — After ANN retrieval of top-50 to top-100 candidates, apply a cross-encoder re-ranker for more accurate relevance scoring. Pass the top-5 to top-10 re-ranked results to the LLM.
The right content exists but is not retrieved. Caused by poor embeddings, wrong chunk boundaries, or the content not being well-represented in the query's vector neighborhood.
Unrelated content scores higher than the right content. Caused by ambiguous queries, overly broad chunks, or embedding model weakness in the domain.
Build a test set of queries with known ground-truth answers to measure and distinguish these failure modes.
Ask the AI assistant about retrieval strategies, choosing k, similarity thresholds, or re-ranking.