ἵστωρ Search Follow the build

What is a context window?

The fixed size of what a model can read at once. It explains most of what people assume is intelligence or laziness in AI answers.

The context window is a model's working memory: everything it can read at once, measured in tokens. Your question, the passages placed in front of it, and the answer being written all share one fixed space, and the standing instructions at the front of it, the system prompt, claim their share first. What does not fit might as well not exist for that answer. Retrieval is how a small window gets used well: only the passages that matter go in, and the rest of the library stays on disk.

The working memory of a model.

When a model answers, it reads a block of text: your question, whatever context was added, and its own answer so far. That block is the context window, and it has a hard ceiling. Text beyond it is invisible, not compressed or remembered. This is why a model can lose the thread of a long chat, why it does not know about documents you never gave it, and why a two-hundred-page PDF cannot be summarized in one gulp. The window is not ignorance; it is a budget.

The window is a budget, not a memory e.g. one 8,192-token window the ceiling read less reliably invisible, not compressed instructions your question a few passages the answer The window is a budget, not a memory e.g. one 8,192-token window instructions your question a few passages the answer read less reliably the ceiling invisible, not compressed
One window, four claimants, and a ceiling that ends it. The standing instructions take their share first, retrieval spends the bulk on a few passages rather than on the whole library, and the answer gets what is left. The veiled band is the middle of that space, where a model reads least reliably, and the region past the ceiling is not compressed or remembered; it is only absent. The split is an example of how a request fills a window rather than a measurement of one.

How retrieval spends the budget well.

This constraint is exactly why retrieval-augmented generation exists. Rather than stuffing a whole library into the window, a retrieval step picks the handful of passages most relevant to your question and places only those in front of the model. The window fills with the right text instead of all the text. On local hardware this matters twice over: a larger window needs more memory and more time for every answer, so the skill is in choosing well rather than in choosing everything.

Why bigger is not automatically better.

Large windows are real and useful, but two honest caveats follow them. First, models attend less reliably in the middle of very long contexts, so what goes in is not all equally read; a wide spreadsheet is the everyday example, and how questions of one work shows retrieval choosing rows instead. Second, memory and speed costs grow with the window, which is the same memory the model itself competes for on a local machine. The practical takeaway for research work is unchanged by either trend: put the right passages in a modest window, verify the claims against them, and treat the context window as the boundary that makes retrieval a discipline rather than a limitation to apologize for. How that boundary is enforced on a request is its own topic: what a token limit is follows the budget from instructions to answer.

Related