What Is Context Rot? Why Long Contexts Degrade and What to Do About It

Context rot is the observed decline in a model's accuracy as its context fills up, long before the stated token limit is reached. Give a model 5,000 tokens of relevant material and it performs well. Give it the same 5,000 tokens buried inside 150,000 tokens of loosely related material and it performs worse, sometimes much worse.

The term became common as context windows grew past the point where anyone could sensibly fill them. It names something people had already noticed: that a bigger window is not the same as a better answer.

Why does it happen?

Three mechanisms, and they compound.

Attention is finite and unevenly distributed

Models do not weight every token equally. Information at the start and end of a long input is used more reliably than material in the middle. This was documented systematically in research on how language models use long contexts, which found performance on retrieval-style tasks degraded significantly when the relevant fact sat in the middle of a long input rather than at either end.

The practical consequence is uncomfortable: where you put something in a long prompt affects whether it gets used. That is not a property anybody wants in a system they are relying on.

Distractors compete with signal

Every irrelevant passage in the window is something the model has to implicitly rule out. Twenty loosely relevant documents compete with the two that matter, and the more plausible the irrelevant material, the more expensive it is to ignore. Independent testing of context rot found accuracy falling as input length rose even on tasks where the relevant information was unambiguous, with the presence of similar-but-wrong content making it worse.

This is why "just retrieve more, to be safe" is such a common and costly mistake. Retrieval recall goes up, answer quality goes down.

Long inputs invite reasoning drift

On tasks requiring multi-step reasoning over a large input, errors early in the chain propagate. Extended benchmarks such as RULER show effective context length, the length at which a model still performs reliably, is frequently far shorter than the advertised window.

That gap between advertised and effective is the heart of the problem. A model rated for a very large window may hold reliable performance across a fraction of it.

The symptoms, and what each one points at

What context rot is not

It is not a bug that gets patched. It is a consequence of how attention works over long sequences. Models improve at it, and the ranking of models on long-context benchmarks changes, but the shape of the problem persists.

It is not solved by a bigger window. This is the most expensive misunderstanding in the space. A larger window raises the ceiling on how much you can include. It does nothing about the degradation curve, and it makes it cheaper to make the mistake of including too much.

It is not the same as running out of context. Hitting the limit is a hard failure with an error. Context rot is a soft failure with a confident wrong answer, which is considerably worse because nothing surfaces it.

How do you know it is happening?

The signature is distinctive once you look for it.

• Answers get worse as conversations get longer, with no error and no obvious cause.
• The model cites something from early in the input and ignores something more relevant from the middle.
• Retrieving more passages made quality drop, which is the clearest single tell.
• The same question answered correctly in a short prompt fails in a long one. This is the cheapest test available, and worth running before any deeper debugging.

Instrumenting for it means logging what was actually placed in the window per request and correlating input size with output quality. That is a core part of LLM observability, and it is generally not visible without it.

What actually fixes it

The fix is not more context. It is less context, chosen better.

Retrieve less, and more precisely

The single highest-value change for most teams. Cutting retrieval from twenty passages to five frequently improves answers while reducing cost and latency at the same time. That combination is rare enough to be worth testing early.

The instinct to retrieve generously comes from a reasonable fear of missing the relevant passage. The way out is precision rather than volume: filter hard before ranking, so what reaches the window is already scoped.

Filter before you rank

Similarity search over an entire corpus ranks everything against the query and hands back the top matches, which is how near-misses end up in the window. Narrowing first, by entity, by owner, by time period, by permission, and only then ranking within that set, produces a much cleaner input.

This is the practical argument for structure over pure similarity, and it is why enterprise knowledge graphs matter here. A graph lets you say "facts about this account, valid now, that this person can see" as a constraint rather than a hope. You are no longer asking the model to ignore the wrong things; you never put them in.

Compact, do not accumulate

Long conversations should be summarized progressively rather than carried verbatim. Most agent frameworks now do this automatically past a threshold. The thing worth checking is what your compaction discards, because a summary that drops the decision made forty turns ago has traded one failure for another.

Put the important things at the edges

Given that position affects use, place the instruction and the most critical context near the start or the end rather than buried. This is a workaround rather than a fix, and it works.

Use memory rather than re-supplying context

If the same background is being rebuilt into the window every session, that is a memory problem being solved with tokens. Agent memory holds distilled facts across sessions, so the window carries a short set of relevant statements instead of the raw material they were derived from. Anthropic's guidance on context engineering makes a similar argument: treat the window as a scarce budget to be curated rather than a container to be filled.

Split the task

A request needing 100,000 tokens of context is often several requests needing 10,000 each. Decomposition avoids the degradation entirely and is usually cheaper.

Context rot and the cost question

The economics point the same way as the quality argument, which is convenient.

Input tokens are billed per request. Filling a large window on every call multiplies spend directly, and unlike output length it is a cost you control completely. A system that stuffs 150,000 tokens into every request to avoid building retrieval is paying continuously to get worse answers.

Teams usually discover this when they instrument per-request cost by feature rather than looking at the aggregate bill. The distribution is where it shows up, since a handful of context-heavy calls can dominate a day's spend. An LLM gateway is where that measurement typically lives.

Where does MCP fit?

MCP is the standard for connecting models to tools and data sources. It governs how context gets in, not how much of it should.

That distinction matters, because connecting more sources is easy and connecting them badly makes context rot worse. Every tool exposed to a model consumes tokens in the window on every request through its definition, whether it gets used or not. Connect fifty tools naively and you have spent a meaningful slice of the window before the user has said anything.

The useful pattern is scoping: expose the tools relevant to the current task, and have the server return targeted results rather than raw documents. A server that answers "who owns this account" with a fact is contributing to a good outcome. One that answers with twelve documents that mention the account is contributing to context rot.

Where Coworker AI fits

This is the problem we build around. Coworker AI connects to 50+ tools and maintains organizational memory across them as a structured graph, so an agent receives scoped, resolved context rather than a pile of retrieved documents. The distinction is between supplying an answer and supplying material from which an answer might be derived.

It exposes that same context over MCP, so the tools your team already uses get the scoped version too. Pro is $29.99 per user per month, Max is $149.99, and Enterprise pricing is on request.

Book a demo if you are hitting this on your own stack.

Frequently asked questions

What is context rot?

The degradation in a model's accuracy as the amount of text in its context window grows, occurring well before the stated token limit. It happens because attention is unevenly distributed across long inputs and irrelevant material competes with relevant material, so more context can produce worse answers.

Does a bigger context window fix context rot?

No. A larger window raises how much you can include, not how reliably the model uses it. Effective context length, where performance stays reliable, is often far shorter than the advertised limit. Larger windows arguably make the problem easier to cause.

Is context rot the same as running out of context?

No, and the difference matters. Exceeding the limit is a hard failure that produces an error. Context rot is a soft failure that produces a confident wrong answer at normal latency with no error, which makes it far harder to detect.

How do I test whether context rot is affecting my system?

Ask the same question twice, once in a short focused prompt containing only relevant material and once in your full production context. If the short version answers correctly and the long one does not, you have it. Then try cutting retrieved passages roughly in half and compare.

Why did retrieving more documents make my answers worse?

Because every irrelevant passage competes for attention with the relevant one, and near-misses are the most expensive kind since they are hard to rule out. Retrieval recall and answer quality are different things, and optimizing the first can damage the second.

How does a knowledge graph help with context rot?

By letting you filter before you rank. A graph can constrain to a specific entity, time period and permission scope before anything is ranked for relevance, so what reaches the window is already narrow. Similarity search over a whole corpus has no such constraint and returns near-misses by design.

Does MCP cause context rot?

MCP governs how models connect to tools, not how much context they consume. It can contribute indirectly, since every exposed tool definition occupies window space on every request and a server returning raw documents adds bulk. Scoping the tool set and returning targeted results rather than documents avoids most of that.

Related reading

• What Is a Context Window?
• What Is AI Agent Memory?
• What Is an Enterprise Knowledge Graph?
• What Is MCP (Model Context Protocol)?
• What Is LLM Observability?
• What Is an LLM Gateway?
• Mem0 vs Zep vs Letta Compared
• LLM Agent Architecture
• AI Agent Orchestration Platforms

The practice built around avoiding this is context engineering.