In a RAG system, the LLM's role is to read retrieved content and synthesize an answer — not to recall facts from training. This is reading comprehension and synthesis, not knowledge retrieval. A smaller, instruction-following model often outperforms a larger one when retrieval quality is good.
For factual question answering, use temperature 0.0–0.3. You want the model to faithfully report what the retrieved context says — not creatively elaborate in ways that introduce inaccuracies. Higher temperature is appropriate only when creativity is explicitly desired (summarization, content generation).
The model attempts to answer using its parametric memory, blending retrieved content with fabricated details. Looks trustworthy, may be wrong.
System prompt says: "If the context does not contain the answer, say so explicitly." Model responds honestly with "I don't have information on this." Trustworthy.
For APIs and automated workflows, instruct the LLM to respond in JSON. A structured RAG response can include the answer text, cited source chunks, and a field indicating when context was insufficient.
Structured output makes post-processing, citation display, and quality logging significantly easier.
Latency — Generation is the slowest step (1–10s). Shorter context and smaller models reduce latency. Responsiveness matters as much as raw capability for user experience.
Logging — Log every query, retrieved chunks, and generated answer. This log is essential for identifying failure modes and measuring improvement over time.
User feedback — Collect thumbs up/down or explicit corrections. Labeled feedback is the most valuable input for improving retrieval and generation quality.
Ask the AI assistant about answer generation, temperature settings, structured output, or production monitoring.