Before you start
You need Python 3.12 or newer for the needlehaystack package, version 2.0.0 on PyPI. Its first demo needs no API key. For a real model, keep an OPENAI_API_KEY or ANTHROPIC_API_KEY in a .env file.
What is context engineering, the first lesson of this course, lists what fills the context on each call. This lesson is about what happens to accuracy as that pile grows.
Measure context rot with a needle test
Start by measuring it on a model you use. A needle test hides one fact, the needle, inside long filler text, the haystack, and asks the model for it. Depth is how far into the context the fact sits, from 0% at the top to 100% at the bottom.
niah, the open-source tool behind the test, runs it over a grid of lengths and depths and writes one result row per cell. The commands below come from its README.
pip install needlehaystack
niah demo --fake # no API key, ~1s, proves the install works
niah reconstruct results.jsonl --row 0
niah demo --provider anthropic # needs ANTHROPIC_API_KEY
niah validate my-run.yaml # parse + resolve, no model calls
niah run my-run.yaml # actually run the sweep, append to JSONL- Install it with
pip install needlehaystack. - Run
niah demo --fake. It sweeps 2 lengths by 3 depths against a fake model inside the process and writesresults.jsonl, so you can check the pipeline without a key. - Rebuild what the model saw with
niah reconstruct results.jsonl --row 0. Each row stores a recipe rather than the full text, and this command turns it back into the exact context. - Put
ANTHROPIC_API_KEYin.envand runniah demo --provider anthropic. The tool loads.envby itself. - Copy the model config from the README into
configs/models/, named after your run config'smodelline (for the example,configs/models/anthropic-opus-4-medium.yaml). Then check both withniah validate my-run.yamland runniah run my-run.yamlfrom the folder that holdsconfigs/.
What a sweep varies
The README's example run config asks for 8 context lengths from 2,000 to 32,000 tokens and 11 depths from 0% to 100%, each with three seeds. Sigmoid spacing packs more of the depths near the top and bottom of the context. The model line names a file under configs/models/ that holds the request settings for the SDK.
Its task, uuid_chain, spreads links such as A to B to C through the context and asks for the value tied to A without showing the chain. The model has to find each hop itself, which tests more than lookup. single, multi and uuid are the simpler built-in tasks.
Point model at what you ship and set context_lengths to the sizes your app sends, so the sweep answers a question about your own prompts.
run_name: "uuid-chain-opus"
model: "anthropic-opus-4-medium" # resolved against configs/models/
task:
type: "uuid_chain"
chain_length: 5
haystack:
type: "files"
path: "PaulGrahamEssays"
sweep:
context_lengths: {min: 2000, max: 32000, num: 8, scale: "linear"}
depth_percents: {min: 0, max: 100, num: 11, scale: "sigmoid"}
seeds: [1, 2, 3]
runner:
concurrency: 2
retries: 2
resume: true
store:
type: "jsonl"
path: "results/uuid-chain-opus.jsonl"The middle of a long input gets lost
Position matters as much as length. In a 2023 study, Nelson Liu and colleagues gave models 20 retrieved documents and a question, with the answer in exactly one document, then moved that document from first place to last.
GPT-3.5-Turbo answered 75.8% correctly with the answer first and 63.2% with it last. At the 10th document it fell to 53.8%, below the 56.1% it scored with no documents at all. Claude 1.3 stayed flatter, between 55.9% and 60.1%. The paper calls the shape U-shaped: best at the edges, worst in the middle.
For a prompt you write, put the fact an answer depends on near the start or the end, not halfway through what you paste.
Source: Hugging Face Papers: Lost in the Middle: How Language Models Use Long Contexts (read November 6, 2026)
Why passing a needle test is not enough
Finding one sentence got easy for most models, so newer tests make it harder. NoLiMa, a 2025 benchmark, writes needles that share no words with the question. To connect them, a model has to know, for example, that the Semper Opera House is in Dresden.
In the paper's latest version, 11 of the 13 models that claim at least 128K tokens fell below half their short-context score at 32K. GPT-4o went from 99.3% to 69.7% (NoLiMa). NVIDIA's RULER found the same gap in 2024. In its latest version, almost all 17 models it tested were near perfect on the plain needle test, yet only half kept satisfactory performance at 32K once tasks needed several needles, tracing or aggregation.
OpenAI's prompting guide for GPT-4.1 reports good needle results all the way to the model's full 1M tokens. It still warns that performance can degrade as more items must be retrieved, or when a task needs the state of the whole context, such as a graph search.
Chroma's July 2025 technical report is titled Context Rot: How Increasing Input Tokens Impacts LLM Performance. Its replication code runs three experiments: needles matched by meaning rather than by words, questions over long chat histories from LongMemEval, and copying a sequence of repeated words. The README sums up the finding: performance varies with input length, even on simple tasks.
The cause lies in how transformers read. Every token attends to every other, which makes n² pairwise relationships for n tokens, so a longer context spreads a fixed attention budget thinner. Models also see more short sequences than long ones in training. Anthropic's context engineering guide describes the result as a gradient, not a cliff.
What to keep out of a long context
In an agent, most of what piles up is tool output and old turns. A June 2026 study of long web-search runs across four open models found that, as accumulated context grew, models more often gave up or gave unsure answers. Removing that context nearly eliminated those outcomes but left more runs unfinished (paper). Cut with a plan, not all at once.
Compaction and memory tools get a lesson of their own, agent memory, and loading tools and files on demand is just-in-time context. The table gives the short version, and those two lessons go deeper on clearing old results, notes and loading on demand.
| What piles up | What to do | Source |
|---|---|---|
| Results of tool calls the agent has finished with | Clear them. Anthropic calls clearing old tool results one of the lightest forms of compaction. | Anthropic: Effective context engineering for AI agents |
| Long tool output | Page, filter or truncate it. Claude Code caps a tool response at 25,000 tokens by default. | Anthropic: Writing effective tools for agents |
| Tools the current task never calls | Connect fewer servers, or load tools when a step needs them. | Anthropic: Effective context engineering for AI agents and Introducing advanced tool use |
| A session that ran through several tasks | Write progress to a file, then start a fresh session that reads it. | Niels Rogge on the Hugging Face blog |
Greg Kamradt's needle test and Niels Rogge's 60% habit
The needle test most long-context reports cite started as a post on X. On 8 November 2023 Greg Kamradt, now president of the ARC Prize, posted his results for GPT-4-128K. He hid one sentence about San Francisco in Paul Graham essays at 15 depths and 15 lengths up to 128K tokens. Recall started to drop above 73K tokens, and facts placed between 7% and 50% deep were missed most often. A fact placed first was found at every length.
His advice then still fits: less context means more accuracy, and no fact is guaranteed to be retrieved. His repository became the niah tool from the first section.
Niels Rogge has added about 50 model architectures to Hugging Face's Transformers library. In a March 2026 article he described porting a video model with OpenAI's Codex app, on GPT-5.3 Codex at the time. The agent read and updated a progress.md file, and he asked it to continue about ten times. Codex's compaction let him stay in one chat. With other tools he restarts, as the quote says. Chroma, whose researchers wrote the context rot report, explains it in a video on its YouTube channel.
Usually, I need to restart a chat everytime the coding agent is hitting at least 60% of its context window, as the model simply becomes dumber after that, a problem known as context rot.
A screenshot of X's own embed of the post, taken on 6 November 2026, with the time in UTC. X cuts the post short with Show more. Select it to open the post on X.
Common mistakes
- Taking a model's advertised window as the length it works well at.
- Testing with a needle that shares words with the question, then expecting the same score on facts worded another way.
- Burying the key fact deep inside a pasted document, below pages of background.
- Keeping one agent session open across unrelated tasks.
- Cutting every bit of history at once, so the agent loses track of where it was.
Try it yourself
- Run
niah demo --fake, thenniah reconstruct results.jsonl --row 0, and find the needle in the rebuilt context. - Copy the run config, point
modelat a model you use and cap the lengths at what your prompts reach in production. - Run the same lengths with
singleand thenmulti, and compare the scores. - Note the input size where your agent's answers start to slip, and plan a restart from notes before it.
Check yourself
In the Lost in the Middle study, where did GPT-3.5-Turbo do worst among 20 documents?
Show answer
B. With the answer near the middle Accuracy fell to 53.8% at the 10th document, below its 56.1% with no documents at all.
Why can a model ace a classic needle test and still miss facts in your documents?
Show answer
A. The needle often shares words with the question, so plain matching finds it NoLiMa removed that word overlap, and 11 of 13 models fell below half their short-context score at 32K tokens.
Which niah command runs a full demo sweep with no API key?
Show answer
C. niah demo --fake The README's quick start runs niah demo --fake against a fake model inside the process.
An agent has worked through three tasks in one session and its answers are slipping. What is the better next move?
Show answer
D. Save progress to a notes file and start a fresh session that reads it A fresh session drops the stale tokens, and the notes file carries forward what the next step needs.
Frequently asked questions
What is context rot in LLMs?
It is the slide in accuracy that comes with longer input, even when everything fits in the window. Because the decline is gradual, no single token count marks where it starts, which is why measuring your own model at your own sizes matters.
Does a 1M-token context window fix context rot?
No, though newer models hold up better. Anthropic reported in February 2026 that Claude Opus 4.6 scored 76% on the 8-needle, 1M-token version of the MRCR v2 test, against 18.5% for Sonnet 4.5. Neither score is perfect.
Where should I put instructions in a long prompt?
OpenAI's GPT-4.1 guide puts them at both the beginning and the end of the provided context, and says that above the context works better than below if you place them once. Anthropic's 2023 long-context prompting post stresses putting instructions at the end, so run your own sweep with both orders.
Is context rot the same as a hallucination?
No, though one can lead to the other. A hallucination is a confident answer that is false. With context rot, the right fact sits in the prompt and the model misses or misuses it. AI hallucinations covers the first.
Sources
- Hugging Face Papers: Lost in the Middle: How Language Models Use Long Contexts https://huggingface.co/papers/2307.03172
- Hugging Face Papers: NoLiMa: Long-Context Evaluation Beyond Literal Matching https://huggingface.co/papers/2502.05167
- Hugging Face Papers: RULER, What's the Real Context Size of Your Long-Context Language Models? https://huggingface.co/papers/2404.06654
- Hugging Face Papers: Diagnosing and Mitigating Context Rot in Long-horizon Search https://huggingface.co/papers/2606.29718
- GitHub: chroma-core/context-rot https://github.com/chroma-core/context-rot
- GitHub: Needle In A Haystack (niah) README https://github.com/gkamradt/needle-in-a-haystack
- PyPI: needlehaystack https://pypi.org/project/needlehaystack/
- Anthropic Engineering: Effective context engineering for AI agents https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents
- Anthropic Engineering: Writing effective tools for agents https://www.anthropic.com/engineering/writing-tools-for-agents
- Anthropic Engineering: Introducing advanced tool use on the Claude Developer Platform https://www.anthropic.com/engineering/advanced-tool-use
- Anthropic: Introducing Claude Opus 4.6 https://www.anthropic.com/news/claude-opus-4-6
- Anthropic: Prompt engineering for Claude's long context window https://www.anthropic.com/news/prompting-long-context
- OpenAI Cookbook: GPT-4.1 prompting guide https://developers.openai.com/cookbook/examples/gpt4-1_prompting_guide
- X: Greg Kamradt, pressure testing GPT-4-128K with long context recall https://x.com/GregKamradt/status/1722386725635580292
- Hugging Face blog: Niels Rogge, How I contributed a new model to the Transformers library using Codex https://huggingface.co/blog/nielsr/contributing-to-transformers-with-codex
- YouTube (Chroma): Context Rot: How Increasing Input Tokens Impacts LLM Performance https://www.youtube.com/watch?v=TUjQuC4ugak