The context window is the model's working memory. Everything visible to the model in a single interaction lives inside this window: your prompt, the conversation history, any documents you provide, and the model's own previous responses.
If information falls outside the context window, the model cannot access it. It does not summarize the overflow — it simply cannot see it.
Context windows are measured in tokens, not words or characters. A rough rule of thumb: 1 token ≈ 4 characters ≈ 0.75 words. A 100-token limit is about 75 words.
System prompt — instructions that define the model's role, tone, and constraints, set by the developer
Conversation history — all previous messages in the session, both user and assistant turns
Current user message — the prompt you just sent
Retrieved context — any documents, data, or retrieved chunks injected alongside the prompt
Model output — the response being generated counts against the window too
The model processes all of this simultaneously. The transformer attention mechanism lets it relate any position in the input to any other position at once.
Equivalent to about 3,000–6,000 words — roughly a short story or a few pages of a technical document.
Early GPT-3 and many production models from 2020–2022 operated in this range.
Sufficient for short Q&A sessions but problematic for long documents or extended chats.
Equivalent to about 95,000–150,000 words — an entire novel or a large codebase.
GPT-4 Turbo, Claude 3, and Gemini 1.5 Pro reached this range in 2024.
Enables processing entire research papers, long conversations, and multi-file code analysis in a single call.
When the conversation grows longer than the context window allows, the oldest content must be dropped to make room. Most chat interfaces and APIs do this silently.
The model does not summarize what it drops. It simply can no longer see that content. This is why long chat sessions sometimes feel like the model "forgot" what you discussed earlier — it literally did.
Long documents must be chunked, summarized, or retrieved selectively before being sent to the model
Chat applications must explicitly manage conversation history, trimming or summarizing old turns
System prompts consume part of the available window — long system prompts leave less room for user content
Output tokens count against the window too — a very long response shrinks available input space
These two concepts are frequently confused. They are entirely different limits.
What the model can see right now, in this specific conversation.
Controlled by what you include in your prompt and conversation history.
You can extend it by pasting information directly into the prompt.
The date when the model stopped being trained — the boundary of what it learned from pretraining.
A model with a 2024 cutoff cannot know about 2025 events from its training alone.
You can work around it by pasting current information into the context window.
RAG is the standard engineering solution for giving LLMs access to large knowledge bases that exceed even the largest context windows. Instead of stuffing all documents into every prompt, a retrieval system finds only the relevant pieces and injects those.
User asks a question — the query enters the system
Retrieval system searches — a vector database or search index finds the most relevant document chunks
Chunks are injected — the retrieved content is added to the context window alongside the question
LLM generates an answer — grounded in the specific retrieved content, not just training data
Answer is returned to user — with citations pointing back to the source documents
RAG lets you build AI systems that reason over millions of documents without requiring a million-token context window or retraining the model. The quality of retrieval determines the quality of the answer — garbage retrieval means garbage response, even with a great model.
Ask the AI if you need help understanding context windows, token limits, the difference between context and knowledge cutoff, RAG, or how to manage long conversations in LLM applications.