More context is not the answer — measured context rot and what to do about it
All 18 models Chroma tested got worse as input grew longer, starting well before the context window was full. What to do about context rot.
On this page
A context window is the total number of tokens a model can take as input at once. “200K” means you can put in up to 200,000 tokens. But the amount you can put in is not the same as the amount the model actually makes good use of. When Chroma measured 18 models, every one of them, without exception, performed worse as input grew longer. The set included not only the then-current GPT-4.1, Claude 4, Gemini 2.5 and Qwen3 but also older models such as GPT-3.5 Turbo. The degradation began well before the context window was full.
Executive summary
“Context engineering” became a hot topic in 2026 not because context windows got bigger, but because measurements confirmed that performance drops if you fill the bigger context window all the way. The practice flipped from “putting more in” to “choosing what to leave out.”
What is context rot?
It is the phenomenon of output quality dropping as input tokens increase. The name took hold with Chroma's July 2025 technical report. One thing to disclose first — Chroma is a company that sells a vector database, and it has a business interest in this topic. It is the source this article cites most, so please read with that in mind. That said, the report publishes its model list and experimental design, so there is comparatively more room for readers to check it.
Let's clear away a common misunderstanding first. This is not a problem of input going over the limit and getting cut off. It happens within the limit, while there is still room. In other words, how much a model “can read” and how much it “can make good use of” are different.
“It's 200K, so up to 200,000 tokens is safe”
The size of the context window is the most input the model accepts without rejecting it, not the point up to which quality holds. The two values are different.
“It passed a long-context benchmark, so it's fine”
Chroma's point is that existing long-context benchmarks do not adequately reflect real usage conditions. The conditions it was measured under matter more than whether it passed.
“A better model will fix it”
Of the 18 models, none was consistently strong across all tasks. Performance depends on the task.
What was measured
Chroma (Kelly Hong · Anton Troynikov · Jeff Huber, 2025-07-14) tested 18 models from Anthropic, OpenAI, Google and Alibaba on three families of tasks. As noted above, Chroma is a company with a commercial interest in this area.
| Task | Design | What it looks at |
|---|---|---|
| Needle in a Haystack NIAH from here on |
8 input lengths × 11 needle positions | The difference between an exact word overlap and a match in meaning only |
| LongMemEval | 306 curated conversational QA prompts, about 113K tokens on average | The gap between giving only the relevant part and giving everything |
| Repeated Words | 7 word pairs; for each pair, 1,090 variants spanning 25–10,000 words | The ability to copy text exactly, not to understand content |
3.1 How much performance drops depends on conditions
The core of this study is that input length alone does not explain the degradation. At the same input length, worse conditions make performance drop further.
Semantic distance between the question and the needle sentence
Embedding cosine similarity is a measure of how close in meaning the question and the answer sentence are. The lower this value, the lower the performance. In the experiments, similarity ranged 0.445–0.775 for essays and 0.521–0.829 for arXiv papers, depending on the source material.
Practical implication: the further apart the question and the answer are in meaning, the worse the result. This is a different axis from whether the words overlap.
Distractors
Performance dropped with even one piece of information that looked similar but was not the answer, and dropped further with four.
In practice, the habit of putting in everything that looks relevant creates exactly this condition.
Model families also differed in how they failed. However, the three points below come from different experiments, so they should not be read together as one trend.
- In the NIAH distractor experiment: the Claude family consistently had the lowest hallucination rate, and the GPT family the highest.
- In the Repeated Words task: a refusal rate of 2.55% was reported for GPT-4.1. This is the value for a single model, not for the whole family.
- In LongMemEval: the gap was especially pronounced for Claude Opus 4 and Sonnet 4, and the cause was conservative abstention (declining to answer).
The most counterintuitive result
This runs against intuition. You would expect a model to understand a well-organized document better, but the measurements showed the opposite.
But do not draw the conclusion “shuffle your documents” from this. It is a phenomenon observed under controlled experimental conditions, and what causes it has not been established. The practical implication to take away is only this — the assumption that “richer context will make it do better” is not supported by measurement. You cannot justify pasting in whole documents as simply “being helpful.”
The monitors rot too
A separate preprint released on May 12, 2026 (Sam Martin · Fabien Roger) looks at the same problem at the monitoring stage, a point where it does particular damage when you run coding agents. It is an independent study, with different authors and tasks from the Chroma study. Its subject is a setup in which an LLM monitors coding agents' transcripts.
Using Opus 4.6, GPT 5.4 and Gemini 3.1 as classifiers, the authors had them find dangerous actions in transcripts longer than 500,000 tokens. The result:
The paper's concluding sentence stings in particular — “Monitor evaluations that do not consider long-context degradation are likely overestimating monitor performance.”
The mitigations the paper tested are re-inserting the instructions at intervals throughout the transcript and having the model provider improve performance through further training. The former can be applied today; the latter is up to the model provider.
This study is an arXiv preprint and has not been peer reviewed.
So what do you do?
Anthropic has published three responses. The shared premise is treating context as a finite budget.
Context compaction
Summarize a conversation that is nearing the limit and restart with the summary as a new context. The skill lies in choosing what to keep and what to discard.
Anthropic's tuning order: first maximize recall so nothing is left out, then trim unnecessary content to raise precision.
Structured note-taking
Keep memory in files outside the context, such as NOTES.md and TODO.txt, and read them back when needed.
The point is to move out of the context whatever does not need to be held all the time. The space is finite, and the more you fill it, the worse things get.
Subagents
Split the work so that each part is explored in depth in its own clean context, and return only a condensed summary to the main agent.
In this structure, the noise of exploration does not pollute the main agent's context.
Running coding agent sessions
What follows from here is practical judgment drawn from the measurements, not measurement results. We have split it in two. One part is what the studies actually tested; the other is practical hypotheses we inferred from their results. The latter are hypotheses, and you need to verify them in your own environment.
7.1 What the studies directly support
| What to do | The measurement behind it |
|---|---|
| Treat input length as a performance variable. Do not assume you are safe because there is room left before the limit | Degradation with length was observed in all 18 models (Chroma) |
| Do not leave the first-pass review of long transcripts to an LLM | Missed dangerous actions increased 2× to 30× after 800,000 tokens of normal activity (Martin · Roger). Note: “narrowing things down first with a mechanical filter helps” is not a mitigation this paper tested — see 7.2 below |
| If you have a long monitoring or review pipeline, remind the model of its instructions partway through | One of the mitigations the same paper tested |
| Design fallbacks to match how a model family fails (retries for abstainers, verification for hallucinators) | The difference in hallucination rates by family in the NIAH distractor experiment, and conservative abstention in LongMemEval (Chroma) |
| Use context compaction, structured note-taking and subagents. Tune the compaction prompt for recall first | Anthropic's official engineering guide |
7.2 What we inferred — unverified hypotheses
| Habit | Hypothesis | Try instead |
|---|---|---|
| Keeping one session going all day | As tokens pile up, the early instructions will in effect be diluted. Note: no study has directly measured the effect of restarting a session | Break at task boundaries, explicitly choose what to carry over, and start fresh |
| Having the agent read every file that “might be relevant” | Files that are similar but not the answer will act like distractors. The distractors in the experiment were built artificially | Give it only what the task needs. If you have to sweep widely, hand it to a subagent and take back only a summary |
| Pasting in whole documents | Length will go up and the share of text that holds the answer will go down. There is no basis for extending the finding that logical coherence hurts to ordinary documents | Put in only the relevant section |
| Handing over a whole long log with “find the problem” | Any method that makes the input shorter will help. The mitigations the paper tested are “periodic reminders” and “better post-training,” not mechanical pre-filtering | Narrow it down first with grep, date ranges and the like before handing it over, and measure the effect yourself |
| Searching and querying with the user's exact wording | Since a large semantic distance between question and answer hurts, refining queries toward the documents' own terms will help. Query rewriting itself was not tested | Measure query rewriting yourself with an A/B test |
The state of the field
Below is a survey of IT and AI infrastructure leaders that Redis published in 2026. It is a survey published by a vendor, and its sample size and sampling method are not disclosed. Redis has a commercial interest in this area. It is right to read it only as the direction of industry sentiment, not as an estimate for the population.
| Response | Share |
|---|---|
| Agree that “AI agents fail more often due to broken context than to broken models” | 73% |
| Say their agents cannot reliably handle relationships across systems | 69% |
| Still at the ad hoc or exploratory stage (Stage 1–2) | 81% |
| Say they have built a compounding context system | 4% |
These figures cannot be used as evidence. The most we can say is that they do not point in a different direction from the measurements in the preceding sections.
Checklist
- Get into the habit of breaking long sessions at task boundaries, whether or not there is room left before the limit
- Move recurring constraints and rules into files (
CLAUDE.md,NOTES.md) - Stop “have it read everything first” and hand over only the scope you need (hypothesis-based)
- Narrow things down mechanically first before pasting (grep, date filters) (hypothesis-based)
- Hand broad exploration to subagents and take back only summaries
- If you use a compaction prompt, recall first, precision later
- If you have LLM-based review or monitoring, treat input length as a performance variable
- Design fallbacks to match how a model family fails (hallucination vs abstention)
What this is not yet
- The figures in this article are not our own measurements. They summarize a Chroma technical report, an arXiv preprint and a Redis survey.
- Chroma is a company with a commercial interest in this area. The report publishes its experimental design and model list, so it is relatively verifiable, but you should read it with that interest in mind.
- The monitor study is a preprint. It has not been peer reviewed, and there are no replication results yet. 2× to 30× is a wide range and varies a great deal with conditions.
- The Redis survey does not disclose its sample or methodology. We cited it only as sentiment, not as evidence.
- What causes “shuffling helps” has not been established. Do not read it as a recommendation to actually shuffle your documents.
- Models keep changing. The Chroma experiments reflect the mid-2025 generation, and the monitor study the first-half-2026 generation. Look at the structural trend rather than the absolute numbers.
Sources
| Source | What we used it for | Link |
|---|---|---|
| Chroma — Context Rot (2025-07-14) Kelly Hong · Anton Troynikov · Jeff Huber. Primary study, though published by a vendor |
18 models, 3 task families, degradation by condition, haystack structure results | trychroma.com |
| Classifier Context Rot (2026-05-12) Sam Martin · Fabien Roger. arXiv preprint, not peer reviewed |
Long-context degradation in coding agent monitoring, 2× to 30× increase in misses | arxiv.org/abs/2605.12366 |
| Anthropic — Effective context engineering for AI agents Primary source. The model provider's official engineering guide |
Compaction, structured note-taking, the subagent structure, and the tuning order | anthropic.com |
| Redis — State of Context Engineering 2026 Vendor survey. Sample size and sampling method not disclosed |
Industry sentiment. Not used as evidence | redis.io |
This document is a technical summary of public materials as of 2026-09-06. The figures will change as model generations change. Before applying any of this in production, check whether each source's experimental conditions match your own environment.