Context rot

Context rot is the systematic degradation of AI performance as the amount of input context grows, even when the underlying task remains simple. It breaks an assumption most people never examine — that a model handles the 10,000th token of a conversation as reliably as the 100th. It does not: measured across increasing input lengths, performance consistently degrades.

The phenomenon matters for any knowledge worker who uses AI for extended conversations, long documents, or multi-step workflows. Surface fluency masks the problem — a rotting context produces plausible-sounding responses that are subtly wrong, with no obvious signal that accuracy has dropped.

The evidence

A 2025 Chroma Research study across 18 LLMs found consistent degradation patterns (Kelly Hong, Anton Troynikov & Jeff Huber, "Context Rot: How Increasing Input Tokens Impacts LLM Performance", Chroma, 14 July 2025 — https://www.trychroma.com/research/context-rot). Its headline result: models "do not use their context uniformly; instead, their performance grows increasingly unreliable as input length grows". That held even on deliberately trivial tasks, where performance degraded "in surprising and non-uniform ways". The report states no overall percentage for the drop — per-model results appear only in its figures. Even a single distractor reduces performance relative to a needle-only baseline, and adding four distractors compounds the degradation further.

A counterintuitive finding from the same research: across all 18 models and needle-haystack configurations, models performed better on shuffled (disordered) haystacks than on logically structured ones — "structural coherence consistently hurts model performance". The report offers no mechanism for this, and says so: structural patterns "could influence how the attention mechanism is applied", but the authors "do not explain the mechanisms behind this performance degradation".

Model failure modes differ by provider. "Claude models consistently exhibit the lowest hallucination rates" — Sonnet 4 and Opus 4 are "particularly conservative and tend to abstain when uncertain, explicitly stating that no answer can be found". "GPT models show the highest rates of hallucination, often generating confident but incorrect responses when distractors are present." On the repeated-words task, Gemini 2.5 Pro showed the most variable output within the Gemini family; the report makes no cross-provider variability ranking.

How it happens

Attention budget depletion. A useful frame from Anthropic's context-engineering guidance rather than from the Chroma report: "LLMs have an 'attention budget' that they draw on when parsing large volumes of context. Every new token introduced depletes this budget by some amount." (Anthropic, "Effective context engineering for AI agents" — https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents)

Signal-to-noise erosion. As conversations accumulate irrelevant logs, stale tool outputs, and deprecated state, the model has to work out what matters before it can use it: "adding irrelevant context adds the additional step of identifying what is relevant, forcing the model to perform two tasks simultaneously". The cost shows up immediately — "even a single distractor reduces performance relative to the baseline (needle only)".

Multi-turn drift. This one is not from the Chroma report, which runs no multi-turn experiment. In "LLMs Get Lost In Multi-Turn Conversation" (Laban et al., 2025 — https://arxiv.org/abs/2505.06120), models "often make assumptions in early turns and prematurely attempt to generate final solutions, on which they overly rely"; and "when LLMs take a wrong turn in a conversation, they get lost and do not recover". Because AI is designed to appear helpful, it will often act as if it remembers things it has actually lost track of.

Position effects are weaker than the folklore. The familiar "lost in the middle" advice does not reproduce in this study. Testing across 11 needle positions, the report finds "no notable variation in performance for this specific NIAH task". The one position effect it does report runs one way only: on the repeated-words task, "accuracy is highest when the unique word is placed near the beginning of the sequence, especially as input length increases". No end-of-context advantage is measured anywhere in the report.

Detection

Warning signs that context rot is affecting a conversation:

The AI contradicts something it said earlier in the same conversation
Responses feel "off" but still sound confident and fluent
Specific numbers, names, or details get changed or dropped
The AI confirms it remembers something it clearly shouldn't
Simple requests that worked earlier in the conversation now produce errors
Hallucination increases on verifiable facts

Two or more of these signs in the same conversation is a reliable signal to refresh context.

Mitigation

Quality over quantity. In the study's LongMemEval tests, replacing a full ~113k-token conversation history with a focused ~300-token context produced significantly higher accuracy across every model tested. The principle: find the smallest possible set of high-signal tokens that enable the desired outcome.

Retrieval-Augmented Generation (RAG). Rather than loading entire documents, use semantic search to pull only relevant chunks into context when needed.

Context compression. Summarize older conversation turns rather than preserving them verbatim. Remove stale tool outputs and prune irrelevant logs as the conversation progresses.

Fresh starts. For long projects, deliberately begin new conversations at natural breakpoints rather than accumulating context indefinitely. Front-load the new conversation with a compressed summary of what matters.

Front-load what matters. On the repeated-words task, accuracy was highest when the target sat near the beginning of the sequence, so put critical instructions and constraints first. Treat this as a weak preference rather than a rule: the needle-in-a-haystack test across 11 positions found no notable position effect at all.

Multi-agent architectures. Distribute work across specialized agents, each maintaining a smaller, focused context. Coordination happens between agents rather than through a single accumulating conversation.

Sources

Kelly Hong, Anton Troynikov & Jeff Huber, "Context Rot: How Increasing Input Tokens Impacts LLM Performance", Chroma, 14 July 2025 — the 18-model study behind every Chroma-attributed result on this page. The report notes that not all 18 models appear in every experiment, because of context-window and thinking-budget constraints.
Laban et al., "LLMs Get Lost In Multi-Turn Conversation", 2025 — the multi-turn drift findings.
Anthropic, "Effective context engineering for AI agents" — the "attention budget" frame.