RAG improvements are incremental and measurable. Each technique addresses a specific class of failures, and their effects can be verified against an evaluation set. Start with measurement — you cannot improve what you cannot measure.
Build an evaluation dataset of 50–200 queries with known correct answers before optimizing. Measure retrieval recall and answer correctness as a baseline. Every optimization should be validated against this set.
Better chunking — Switch from fixed-size to recursive or semantic chunking. Prepend document titles and section headings to chunks so isolated passages are interpretable. The most common quick win.
Hybrid search — Add BM25 keyword search alongside vector search. Merge results with Reciprocal Rank Fusion. Consistently outperforms pure vector search, especially for queries with specific names or identifiers.
Re-ranking — After ANN retrieval of top-50 to top-100 candidates, apply a cross-encoder re-ranker (Cohere Rerank, BGE-Reranker). Significantly improves final retrieval precision.
Metadata filters — Scope retrieval to relevant subsets by product, category, date, or access level. Improves precision and reduces search latency simultaneously.
Query rewriting — Use the LLM to expand ambiguous or short queries before embedding. Multi-query: generate several phrasings, retrieve for each, merge results to broaden coverage.
Better prompts — Explicit faithfulness instructions, format requirements, and edge-case handling reduce hallucination and improve answer quality without changing the retrieval pipeline.
Fix chunking first (most impactful, cheapest). Add hybrid search next. Add re-ranking if precision is still insufficient. Add query rewriting for ambiguous query patterns. Apply prompt improvements throughout. Measure after each change.
Ask the AI assistant about any of these improvement techniques — hybrid search, re-ranking, query rewriting, or building a RAG evaluation set.