We had a support-ticket triage system that needed to reference a growing knowledge base — a few hundred internal docs, some of them long. The tempting move, once 200K+ token context windows became normal, was to just stuff the whole knowledge base into the prompt and skip building a retrieval pipeline. I tried it. It technically worked. It was also slower, more expensive per call by a wide margin, and — this is the part that actually mattered — noticeably worse at picking up a detail buried in the middle of the context compared to one near the start or end.

This isn't a secret at this point, people call it the "lost in the middle" effect, but living through it in a real pipeline is different from reading about it in a paper. We ran our own version of a needle-in-a-haystack test using our actual docs instead of synthetic ones, and the drop-off in the middle third of a long context was consistent enough across three different models that I stopped trusting "just put it all in context" as a default strategy for anything where a wrong answer has a real cost.

What actually worked better

A boring hybrid: retrieve a focused set of relevant chunks with a decent embedding model, keep the context small — under 20K tokens in our case — and let the model reason over a curated set instead of a dump. Latency dropped by more than half. Cost dropped by more. And accuracy on our internal eval set went up, not down, because the model wasn't competing with irrelevant context for attention.

Where large context windows genuinely earned their keep for us was different: long-document summarization where you actually need the whole document present because you can't know in advance which parts matter, and multi-turn agent sessions where the context is the conversation history itself, not a retrievable corpus. Those are real use cases. "I don't want to build a retrieval pipeline" is not one, even though it's the one the marketing implicitly encourages.

A test worth running before you decide

  • Take five real questions your system needs to answer well.
  • Bury the answer at different positions in a long context — start, middle, end.
  • Compare accuracy against a version with a tight, retrieved context of just the relevant passage.

If the tight-context version wins or ties, you don't have a context-window problem, and a bigger model won't fix what's actually a retrieval quality problem. This test takes an afternoon and will save you from a much more expensive mistake.