From 4K to 1 Million Tokens: How the Context Window Evolved and Why More Context Does Not Mean Better Memory

The Number That Changed the Market: What a Context Window Is—and What It Does Not Measure

A context window is the token budget a model can consider during a single invocation. In practical terms, it usually includes instructions, conversation history, attached or retrieved documents, tool calls and tool results that are fed back into the exchange, and, depending on the interface, the output that is yet to be generated. A context figure should therefore not automatically be read as space available for documents: part of the budget may already be occupied before the task begins.

A specification of 128K, 200K, or 1 million tokens primarily describes an admission limit or a supported configuration. It is an important property: it can avoid splitting extensive material at the outset and may make it possible to keep evidence, instructions, and traceability in a single request. Yet it does not by itself establish that the system will respond with equal fidelity regardless of position, resolve documentary contradictions, or turn a relevant passage into a correct decision.

It is also important to distinguish context from memory in the broader sense. Context is information supplied in the current interaction. Persistent memory requires storing, selecting, updating, and governing information across sessions or tasks. A model may receive a complete case file and still lack a reliable mechanism for determining which fact should persist, which version takes precedence, or when an earlier preference is no longer valid. This distinction is central when designing document assistants and agents.

Expanding context is a useful input capability, not a general guarantee of understanding. The operational question is not which published number is highest, but whether, for a specific distribution of documents and decisions, including more material improves verifiable accuracy without exceeding cost and latency limits.

A Useful Timeline: From Dense Attention to Context Extension

The original Transformer established attention as a mechanism for relating positions within a sequence and used positional encodings to represent order. Its full-attention formulation is powerful, but comparing many positions with one another creates compute and memory pressure that grows rapidly with sequence length. In early practical uses of language models, relatively short windows were not merely a product choice: they reflected training and inference constraints.

Subsequent progress did not follow a single path. On one hand, implementation improvements reduced data movement between memory and processor without changing the mathematical result of exact attention. FlashAttention is a representative milestone on this path: it reorganizes computation around the memory hierarchy. It does not by itself eliminate the growth associated with dense attention, but it can make lengths or batch sizes feasible that were less practical with earlier implementations.

On the other hand, positional representations became a decisive part of extension. RoPE encodes position through rotations; later work proposed interpolating positions to adapt RoPE-based models to larger windows through limited fine-tuning. This mechanism is not equivalent to demonstrating uniform use of every position: it changes how distance and order are presented to the model, while its eventual behavior also depends on data, training, and task.

A third path treats context as a stream rather than as a block that must remain intact in the cache. Streaming-attention mechanisms with attention sinks propose retaining a small set of attention states together with recent tokens. They are relevant to prolonged interactions, but they change the problem: a retention policy determines what remains available and what is discarded. They do not constitute infallible semantic memory.

Finally, providers have exposed context windows of 1 million tokens or more in some model families. Gemini documentation presents these capabilities together with caching options and cost and latency considerations. This is evidence of an interface and product available under specific conditions; it should not be turned into an independent demonstration of reliable reasoning over any million-token input.

Four Layers That Must Not Be Confused

The first layer is admission. A system admits an input when it tokenizes it and accepts it within its limit. The second is effective processing under an operating budget: the same input may require a lengthy prefill, consume memory capacity, or reduce available concurrency. Two systems that admit the same volume can behave differently in response time and cost per task.

The third layer is information retrieval. Here, the question is whether the model locates a specific fact, clause, date, or relationship when it is distributed across documents and surrounded by plausible but irrelevant material. Research on the phenomenon known as lost in the middle evaluated both multi-document question answering and key-value retrieval and observed performance variation according to the position of relevant information. This evidence argues for measuring positions, not only averages.

The fourth layer is coherent use of evidence. A model may cite or extract a correct passage and then produce a synthesis that combines incompatible versions, violates a priority rule, or takes an action not justified by the source. This layer requires decision and generation tasks with explicit criteria, not merely a text-search test.

These layers also help avoid a common error in comparisons: turning an API input limit into a global capability ranking. When comparing options in the comparison route, it is better to record maximum input, maximum output, modality, measured task performance, and measurement conditions separately. The nominal figure is an attribute; reliability is an empirical outcome.

What Changes at Inference Time: Prefill, Generation, KV Cache, and Concurrency

Long inference has at least two phases with different profiles. During prefill, the system processes input tokens to construct the states needed to continue generation. During decoding, it generates new tokens incrementally and reuses those states. A long input can concentrate a substantial portion of initial waiting time even when the final answer is brief; a long output then adds its own duration.

The key-value cache, or KV cache, avoids recalculating the representations of previous tokens for every generated token. It is essential for efficient generation, but it uses memory, and its size grows with attended length, architecture, precision, and the number of simultaneous requests. Work on asymmetric two-bit quantization for the KV cache identifies that cache as a memory bottleneck, especially as context and batch size grow. Quantization can relieve that pressure, but it introduces an additional choice involving quality, compatibility, and evaluation.

Actual cost is not a flat rate derived only from multiplying tokens by a price. Prefill, output length, context reuse or caching where available, retries, number of turns, concurrency, and reserved capacity all matter. Gemini long-context documentation notes specific latency and pricing considerations; those conditions should be checked in the current documentation and under the organization’s own workload.

An evaluation should therefore report a distribution, not merely an average. The average can conceal cases in which long inputs block resources or materially raise the 95th-percentile latency. It should also separate context-preparation time, time to first token, and completion time, because each measurement suggests a different mitigation.

Why Simple Tests Fail

A demonstration in which the answer appears at the beginning or end of a clean document does not represent most real repositories. Relevant evidence may sit in middle positions, inside a table, in an earlier version that has been superseded, or across sources that use different terminology. Repeating the same question at many locations can reveal positional degradation that a single test does not expose.

Distractors must be plausible. Adding random text primarily measures robustness to easy noise; adding similar policies, old figures, or nearly identical clauses measures the ability to resolve ambiguity. It is also useful to introduce controlled contradictions and define the resolution rule in advance: for example, the most recent approved version prevails, or the source designated as authoritative prevails. Without a gold rule, a failure cannot be attributed to the model.

Agents add another source of pressure: context competes with tool descriptions, search results, execution states, and security messages. A larger window may reduce the need to trim, but it can also make obsolete or irrelevant information continue to exert influence. The design should limit which results are reintroduced and preserve provenance identifiers so that an action can be reviewed afterward.

It is not enough to ask a model to claim that it used a source. Output should include internal references to stable fragments of the frozen corpus, and an evaluator should verify that they support the answer. Traceability does not eliminate hallucinations or ensure that an inference is valid, but it turns a claim into a reviewable object.

Long Context versus RAG, Summarization, and Persistent Memory

Long context, retrieval-augmented generation—RAG—summaries, and persistent memory are complementary patterns, not steps on one scale. Long context keeps more literal material in one call. RAG selects a subset from an index or repository. Summarization compresses information, at the cost of potentially losing detail. Persistent memory maintains data across interactions through policies for writing, updating, expiration, and access.

RAG is well suited when the repository exceeds the window, changes frequently, or requires filtering by permissions, date, entity, or jurisdiction. It also reduces the amount of text that must be processed on every turn. Its risks shift to indexing, recall, ranking, and the loss of relationships between fragments. Long context may be preferable when a task depends on comparing many parts of a bounded set, provided tests demonstrate a benefit over well-configured selection.

Summaries help preserve continuity, but they should not be treated as a primary source when a task requires literal precision. A prudent architecture preserves links between the summary and source fragments, allows returning to them, and distinguishes extracted facts, interpretations, and decisions. Persistent memory requires even more governance: who can write to it, what can be forgotten, how it is corrected, and which data must not persist.

The choice should start with an organization’s own evidence. In the discovery route, teams can identify the documents, tools, and constraints that characterize the workflow; in the learning route, they can establish definitions and criteria; and in the comparison route, they can contrast results under the same corpus and budget. The default architecture should not be decided by advertised length.

Your Own Evaluation Protocol: From Demonstration to Decision

A minimum protocol begins with a frozen, documented corpus. It should include representative formats and lengths, versions, permitted metadata, and a clear separation between development and final evaluation. For each task, define an expected answer, the supporting evidence, the rule for resolving conflicts, and the risk level of an incorrect answer. If no single answer exists, the criterion should allow for uncertainty or human escalation.

Next, distribute relevant evidence across multiple positions: beginning, middle, and end. Vary the distance between pieces that must be combined and add semantically close distractors. Evaluate at least literal extraction, multi-document answering, contradiction resolution, and a constrained decision or action. Metrics should distinguish citation fidelity, decision accuracy, appropriate abstention rate, cost per correct case, and latency percentiles.

Compare configurations that consume similar budgets: full context, RAG, RAG plus neighboring documents, summarization with return to source, and, where applicable, long context with caching. Keep the model, instructions, and evaluator fixed when the aim is to isolate architecture. When the model changes, report that change as an additional variable and avoid attributing the entire effect to length.

Before adopting a solution, set explicit thresholds. For example, an improvement should exceed a defined margin in correct decisions and evidence fidelity, should not worsen p95 beyond the service limit, and should remain within a maximum cost per valid task. Specific values depend on the use case; they cannot be inferred from a public context specification.

How to Read 128K, 200K, or 1M Tokens Without Promising Unlimited Understanding

A responsible specification should be read alongside five questions: what the maximum input is, what the maximum output is, which modalities it accepts, which pricing and latency conditions apply, and what behavior has been measured on the task of interest. Input and output are not interchangeable: reserving a large output can reduce space available for documents, and a task with a short answer can still experience substantial prefill delay.

The date and documentation version also matter. Limits, models, modalities, and caching policies can change. In an internal record, it is useful to note the consultation date, the exact model or service identifier, and relevant conditions rather than retaining only a number that may soon be outdated.

The conclusion is not that long windows are useless. They are a meaningful technical expansion and can simplify tasks that previously required aggressive chunking. The conclusion is narrower: usefulness must be demonstrated on the real document distribution, with traceable evidence and within an acceptable operating envelope. More available tokens can improve an application; more tokens without selection, evaluation, or governance can increase cost and the surface area for error.

For product teams, the practical decision is to treat context as a measurable budget. Send more information when it demonstrably improves retrieval and decisions; retrieve, summarize, ask for clarification, or escalate when those options offer better evidence and control. In this way, a 1-million-token window stops being an abstract promise and becomes an evaluable technical option.

Keep exploring

Corrections and transparency

If you spot incorrect or outdated information, send us a correction with the page and source we should review.