RAG substantially improves LLM applications but introduces its own failure modes. The most dangerous failures are silent — a confident-sounding wrong answer rather than an honest "I don't know."
Bad retrieval → bad answers — If the right content is not retrieved, the model cannot answer correctly. Caused by: document never ingested, poor parsing, bad chunking boundaries, weak embeddings in the domain, or ambiguous queries.
Poor chunking loses context — A retrieved chunk contains the answer but the surrounding context that makes it interpretable was cut off. "The limit is 100" without what the limit applies to is useless.
Model still hallucinates — The LLM can misread retrieved content, blend it with parametric memory, or infer beyond what the text states. Strong system prompt instructions reduce but do not eliminate this.
Context window overflow — Too many or too long chunks overflow the model's limit. Truncated content may be the most relevant part. Long contexts also cause the "lost in the middle" attention problem.
Deleted or updated documents must have their vectors actively maintained. A stale knowledge base returns outdated information confidently — often worse than admitting uncertainty.
Query embedding + retrieval adds 100–400ms overhead. Small for most applications, but potentially unacceptable for strict real-time requirements.
RAG quality is difficult to measure automatically. Build a test set of queries with gold-standard answers and measure retrieval recall, answer faithfulness, and answer correctness. You can only systematically improve what you can measure.
Ask the AI assistant about RAG failure modes, how to diagnose retrieval problems, or how to build RAG evaluation infrastructure.