Your context window is a spec, not working memory

A 128K context window is a spec sheet number, not working memory. Every "it reads your whole codebase" demo should ship with an ablation: same question, same documents, answer planted at 10%, 50%, and 90% depth.

The canonical result is Lost in the Middle (Stanford/Berkeley, TACL 2024): accuracy follows a U-shape. Answers at the start or end of context? Fine. Buried in the middle, GPT-3.5-Turbo did worse than with no documents at all — over 20 points lost to placement alone.

The industry's favorite proof, the needle-in-a-haystack heatmap, is the lenient version of this test. One needle, literal keyword overlap, GPT-4 grading GPT-4, roughly $200 in API calls (Kamradt, 2023). Real work is never a single planted sentence in a pile of essays.

Hard the test up and claimed context halves. RULER ran 17 models through 13 harder tasks: GPT-4 held up to 64K of a claimed 128K, Llama3.1-70B the same, and LWM-7B managed under 4K despite claiming a million tokens.

Still true in 2025. NoLiMa stripped literal keyword matching and 11 of 13 models fell below half their short-context baseline at 32K. Context Rot showed 18 models — including current frontier ones — degrading on trivially simple tasks as input grows.

The model is not "distracted" or "overwhelmed." Distraction is a human story. This is a positional accuracy curve in the network: measurable, plottable, ablatable. No incense required.

My opinion: treat the context number like a car's top speed — technically true, useless for your commute. Put critical documents first or last, retrieve instead of dumping, and eval at your real lengths before you ship. The spec says what fits. Only your tests say what works.

Sources

• Lost in the Middle: How Language Models Use Long Contexts (arXiv:2307.03172)
• gkamradt/LLMTest_NeedleInAHaystack (README)
• RULER: What's the Real Context Size of Your Long-Context Language Models? (arXiv:2404.06654v3)
• NoLiMa: Long-Context Evaluation Beyond Literal Matching (arXiv:2502.05167)
• Context Rot: How Increasing Input Tokens Impacts LLM Performance (Chroma, July 14, 2025)