October 29, 2026

Is RAG dead now that models have long context?

Adam Cogan AI, General AI, Long context, RAG, Software Architecture 0 Comments

In the old days of AI (not that long ago! 🤣), models had tiny 4K-token context windows. You couldn’t give an LLM your entire knowledge base, so Retrieval-Augmented Generation (RAG) was essential. It had one job, find the most relevant information and add it to the prompt.

This gave the model access to private, recent, or specialised data it wasn’t trained on, without retraining it.

What changed?

Context windows grew from thousands to millions of tokens, while prompt caching made repeated context faster and cheaper. Devs can now place an entire codebase or document collection straight into the prompt. If the model can see everything at once, why retrieve only a few chunks?

That’s why people started saying “RAG is dead”. Sometimes long context is the simpler choice. It works well when your data fits in the window, and the answer depends on seeing it all together.

Why RAG isn’t dead

Long context is useful, but it doesn’t replace RAG. Sending everything can increase cost and latency, although caching can reduce both. Irrelevant information can also distract the model. When sources change, the supplied context needs refreshing and some cached content may need rebuilding. RAG indexes also need to stay up-to-date. Sending everything in one prompt is no longer an option once your data outgrows the context window.

At SSW, we touched on this on a project called Beryl, an AI payroll agent built for the Australian Payroll Association (APA). Payroll rules change often and sit inside thick award tables, and even official government sites can lag behind award changes. We also built a daily crawler that spots published updates and triggers re-indexing, so changes flow in overnight without anyone updating a PDF. A crawler can only pick up changes once the source publishes them. Beryl answers thousands of questions a day, with over 95% accuracy.

RAG keeps the prompt small by retrieving material relevant to each question. But retrieval can miss a key exception or lose context when documents are split into chunks, so it needs testing too. When configured correctly, the retrieval layer can also enforce access controls before information reaches the model, so people only get results they’re allowed to see.

Many real systems benefit from a mix:

Use long context for compact, reusable data.
Use RAG with hybrid search when you’re searching a large collection and need both keyword and semantic matching.
Use agentic search when one result leads to the next query, or the agent needs to check a live source. MCP is one way to connect those sources.

Start with your data, not the architecture. Give the model the smallest, freshest context that gets you a good answer. Test answer quality, cost, and latency on representative questions before settling on an approach.