Context Window

The window exists because attention compares every position against the others, so the computational and memory cost of a request grows steeply with length. That cost is why a longer window is not free even when it is available: a request carrying a large context is slower to process and more expensive, and, because the key-value cache scales with sequence length, it consumes more of a serving instance’s memory.

The practical consequence in agents and chat applications is that context is a budget to be managed, not a container to be filled. Long-running conversations need a compaction strategy — summarise older turns, keep the system prompt and the recent window intact, and re-retrieve details on demand instead of carrying them forever. Verbose tool output is the usual culprit when a previously reliable agent starts ignoring its instructions: the instructions are still there, but they are competing with thousands of tokens of API response.

The misconception is that a large window makes selection unnecessary. Capacity is not attention. Filling a window with loosely relevant material measurably degrades answers compared with supplying a smaller, well-chosen set of passages — and you pay for the privilege on every request. Treat the window as headroom for the cases that genuinely need it, not as a default to be consumed.