There are two ways to give a model knowledge it was not trained on: paste everything into the context, or build retrieval that fetches only the relevant piece. Both work. The economics and the failure modes are not the same.
Two different mechanisms
Long context is just a bigger prompt. You take your documents, concatenate them, and send the whole thing to the model along with the question. The model decides for itself what is relevant.
RAG, retrieval-augmented generation, works differently. Text is chunked ahead of time and turned into embeddings, vector representations of meaning. A user question triggers a search for the nearest chunks by meaning, and only that subset lands in the prompt, not the whole corpus.
The difference looks small on paper. On the bill and on answer quality, it is not.
What happens to the bill
With long context, every call pays for the entire corpus as input tokens, even if the question touches one paragraph out of a thousand pages. As the corpus grows, the price of every call grows with it, linearly and with no ceiling.
Prompt caching softens this if the same large context gets reused within the cache window: you pay full price once, then input is cheaper. If questions vary and the corpus keeps changing, caching does not save you.
With RAG, the price per call does not depend on corpus size, it depends on the size of the slice you decide to hand the model. The knowledge base can grow to a million documents and the prompt still gets a handful of chunks. The cost of that is separate infrastructure: a vector store, an embeddings pipeline, sometimes an extra model call to rerank candidates.
What happens to accuracy
Long context has a known weakness: models are worse at holding on to details buried in the middle of a large text than at the start or the end. The longer the context, the higher the risk that the fact you need gets lost in the noise.
RAG has a different weakness. Answer accuracy is bounded by search quality. If chunking is coarse, the relevant fragment might simply not make it into the retrieved set, and the model will answer confidently and wrong, because it never saw the data at all. That is a quiet failure: no log, no apology, just the wrong answer.
What breaks on call
Long context is simple by construction: there is no separate pipeline, nothing to page you at 3am. The price of that simplicity is that you manage the token budget yourself, and sooner or later you hit the model's context window if the corpus keeps growing.
RAG is a whole system. You pick a chunking strategy, pin an embeddings model version, rebuild the index on every content update, and watch whether search is actually finding relevant material. Any one of those parts can quietly degrade, and you usually notice through user complaints about odd answers rather than through an alert.
The hybrid most teams end up with
In practice teams rarely pick one approach cleanly. A small, stable corpus, a single project's docs, a team knowledge base, usually just goes into context whole, especially with caching available.
A growing or varied corpus usually moves to RAG, with one adjustment: search retrieves whole relevant documents, not short snippets, and those full documents go into context afterward. That gets you retrieval precision plus the model's ability to reason over complete text instead of a torn-out sentence.
A real-world example: a support bot built on five help articles does perfectly well on long context, all five fit comfortably in the prompt. The same bot on three hundred pages of API documentation, where the authentication section changes weekly, usually moves to search, because the whole corpus no longer fits without the model losing track of the details.
How to decide for your own case
Three questions settle most cases. How big is the corpus and how fast does it grow? If the whole thing fits comfortably in context and barely changes, you do not need search. How often does the data change? If content updates daily, you will be rebuilding the index just as often, and that costs money and engineering time. Do you need one exact fact or synthesis across many sources? An exact fact is usually better found and quoted through search; synthesis over a large body of material is often easier on full context.
Pricing the difference is more concrete than it sounds: the Claudexia calculator lets you estimate the cost of a call at different context sizes and compare it against the pair of calls RAG usually needs, reranking included.
In short
Long context is simpler and more accurate at moderate volume, but the bill scales with the corpus. RAG keeps the bill flat at any data volume but adds infrastructure and the risk of missing the right chunk. Small, stable corpus: context. Growing and varied: search that hands the model whole documents, not fragments.