A RAG system operates in two distinct phases: an offline indexing phase that builds the knowledge base, and an online query phase that retrieves and generates answers for each request.
Collect documents — Load from PDFs, web pages, databases, wikis, or APIs.
Parse and clean — Extract clean text, remove formatting artifacts, headers, and boilerplate.
Chunk — Split into pieces of 200–1000 tokens with overlap between adjacent chunks.
Embed — Pass each chunk through the embedding model to produce a vector.
Store — Save the vector, original chunk text, and metadata (source, date, category) in the vector database.
Embed the query — Convert the user's question into a vector using the same embedding model used for indexing.
Retrieve — Search the vector database for the k most similar chunks (typically 3–10). Apply metadata filters if needed.
Inject context — Assemble retrieved chunks into a context block and insert into the LLM prompt alongside the user's question.
Generate — The LLM reads the full prompt and generates a grounded answer, citing sources if applicable.
RAG adds roughly 100–400ms of overhead (query embedding + retrieval) relative to a direct LLM call. This is small compared to LLM generation time (1–10 seconds) and imperceptible to users in most applications.
Keeping the knowledge base fresh requires a continuous ingestion pipeline: detecting when source documents change, re-processing updates, and removing stale vectors. This is often the most operationally complex part of a production RAG deployment.
Ask the AI assistant about the RAG workflow, the offline vs online phases, or latency considerations.