RAG cost: what a query actually spends, counted not estimated

“How much does RAG cost?” gets answered with a rule of thumb or a pricing calculator that asks you to guess your token counts. The token counts are the whole question, so guessing them is guessing the answer.

We instrumented a local RAG pipeline so that every token entering and leaving the model is counted, under a fixed seed and a published prompt. Figures from our cost and latency study.

k is the single biggest dial

Widening k from 3 to 8, with chunking held identical, multiplied prompt tokens by 2.298.

Scaling the context linearly with k would predict 8/3. The measured figure is 2.298 on the whole prompt and 2.64 on the retrieved context alone. The fixed template and the question dilute the ratio, which is why the whole-prompt number is smaller than the context number.

In absolute terms, config C sends 1,117.8 prompt tokens per query where config B sends 486.4.

Across this matrix, with the model, the prompt and the corpus held fixed, k moved prompt tokens further than any other change measured. If you are billed per input token, that is the dial to check first.

Where the prompt tokens go

config	stack	prompt tokens (mean)	context share	template share
B	k=3, 1,000/200	486.4	79.2% (9,628/12,160)	17.3% (2,100/12,160)
C	k=8, 1,000/200	1,117.8	90.9% (25,413/27,945)	7.5% (2,100/27,945)
D	k=8, 512/128	759.9	86.7% (16,466/18,998)	11.1% (2,100/18,998)

The template tax is fixed, so its share is not

The instruction scaffolding costs the same 84 tokens on every query, whatever else changes. That is the same absolute number in every row above.

Because it is fixed, its share moves with everything else: 17.3% of the prompt at k=3, falling to 7.5% at k=8. Nothing about the template changed. The denominator did.

This matters when someone tells you prompt engineering will cut your bill. On a short-context configuration the template is worth attacking. On a wide-k one it is already noise.

Output tokens are smaller and more compressible than you think

Under config B, completions averaged 12.9 tokens, 323 across 25 answers. Of those:

• the answer itself: 48.6% (157/323)
• the citation protocol line: 36.5% (118/323)

The citation line is worth paying for, because it feeds the guardrail. But it should be priced as protocol rather than counted as answer, and on a per-query basis it is a third of the output.

There is also a wrapper overhead the chat template adds: 29 tokens per request, identical on every request in this run.

Pricing it on your provider

Our dollar cost for this run is 0, because every call ran locally on hardware already paid for. That zero is not a claim that RAG is cheap and it is not your number. What transfers is the token counts and the method.

Apply your provider’s rates to the counts:

cost_per_1k_queries = 1000 * (prompt_tokens_mean * R_in + completion_tokens_mean * R_out) / 1e6

with R_in and R_out in dollars per million tokens.

What we refused to publish

Latency. This machine ran a second workload on the same GPU, so the runner checks for contention before timing and refuses to record wall-clock when it finds any. Token counts are unaffected, because they are deterministic given model, text and seed. A latency number measured under contention would have been a number, and it would have been wrong.

The full accounting, the configuration matrix and the code archive are on the cost and latency page.

Related: what BM25 bought on the same stack, and how we evaluate RAG.