Where Your RAG Pipeline Actually Spends Money

Ask most teams where their RAG pipeline's cost comes from and they'll point at the generation call — the final prompt sent to the model with retrieved context attached. That's often the visible cost, but it's rarely the whole story. A RAG pipeline bills at three separate stages, and the balance between them shifts a lot depending on how the system is built.

Stage 1: Embedding — usually cheap per call, expensive at re-index scale

Embedding models are priced far below generation models — often a fraction of a cent per thousand tokens. For steady-state ingestion of new documents, this stage is rarely the bottleneck. It becomes expensive at re-index time: switching embedding models, changing chunk size, or rebuilding an index after a schema change means re-embedding the entire corpus at once, which can turn a normally negligible line item into a five-figure one-time cost if the corpus is large.

Stage 2: Retrieval — not an API cost, but not free either

Vector search itself typically isn't billed per query the way an LLM call is, but it's not cost-free — it's infrastructure cost (hosted vector database pricing, compute for hybrid search or re-ranking models) that lives outside the AI provider's bill entirely. This is the piece most cost dashboards miss, because it doesn't show up on an OpenAI or Anthropic invoice — it shows up on a separate infrastructure bill that rarely gets connected to the same conversation.

Stage 3: Generation — where chunk strategy determines the bill

This is where most of the controllable cost actually concentrates, and it's driven almost entirely by how much retrieved context gets stuffed into the prompt. Retrieving the top 20 chunks 'just to be safe' instead of the top 5 doesn't meaningfully improve answer quality in most cases, but it does multiply input tokens on every single query — and unlike embedding, generation is billed at the expensive end of the pricing table.

The levers that actually move the total

• Chunk size and count — smaller, more targeted chunks retrieved in smaller numbers usually cut generation cost more than any other single change.
• Re-ranking before generation — an inexpensive re-ranking step that trims 20 retrieved chunks down to the 5 that actually matter moves cost out of the expensive generation stage.
• Caching repeated context — if the same document chunks get retrieved across many queries (a common FAQ, a frequently referenced policy), cache that portion of the prompt rather than re-sending it fresh every time.
• Separating embedding cost from generation cost in reporting — treating RAG as a single line item hides which stage is actually driving a spend increase.

The practical fix isn't a single optimization — it's visibility into which stage is actually growing. A pipeline where generation cost triples while embedding stays flat points at retrieval tuning (too many chunks, too large a context). A pipeline where embedding suddenly spikes points at a re-index event, not a usage pattern change. Without cost broken out by stage, both look identical: 'the RAG feature got more expensive,' with no next step.