What a RAG query costs in tokens, measured not estimated

Every figure below is produced from the results file by the renderer that writes this page, not typed into it. That is a different guarantee from the sister audits, which evaluate expressions against the measurements file at build time, and it is worth being exact about what each half of it covers. tests/test_report_reproduces.py fails if this page is not byte-for-byte what the renderer produces from the committed results, which catches a hand-edited or stale page and nothing else: it cannot see a figure the renderer itself invents. That gap is real and was found in review, so it has its own check. test_every_number_in_the_report_comes_from_the_results_file requires every numeral in this page and the report to appear in the run they were rendered from, which catches an invented numeral that matches no recorded value. It is not a proof of provenance: the vocabulary admits several renderings of every recorded number and every digit run inside every string, so a wrong figure that collides with some unrelated value passes, and a number written as a word is invisible to it. It caught the invented one that was here.

Cost optimisation advice for LLM pipelines usually arrives as folklore: shrink your chunks, lower your k, cache your prompts. This piece measures instead. One local RAG stack, the same one our failure audit broke on purpose, instrumented so that every token entering and leaving the model is counted, under a fixed seed, a temperature pinned to zero by the client that made every request (costlat/client.py in the archive), and a published prompt.

One honest complication shaped the whole project. This machine runs a second automated workload on the same GPU. Token counts do not care about that: they are deterministic given the model, the text and the seed, whoever else is in the queue. Wall-clock does care, so the timing harness checks for contention before and after every sample and refuses to record when the box is shared. The refusal is printed, logged and written into the results file. We think declining to measure is a better look than publishing a number that measures a queue.

The meter is checked before anything is metered

Token counting here means asking the model runtime itself: Ollama reports prompt_eval_count for exactly the text submitted when no chat template is applied. That behaviour is version-dependent, so the run opens with five instrument controls: identical text counts identically twice, longer text counts strictly more, parts nearly sum to wholes (the residual is measured and carried, not assumed zero), embedding counts grow monotonically below the context limit, and the one-token generation cap actually caps. On this run: all controls passed.

control	verdict	detail
C1_count_stability	pass	first 210, second 210, cache_bypassed True
C2_suffix_sensitivity	pass	base 210, with_tail 218
C3_near_additivity	pass	part1 21, part2 11, whole 32, residual 0
C4_embed_count_monotonic	pass	tokens_400_chars 68, tokens_900_chars 149
C5_num_predict_1_limits	pass	eval_count 1

Read C3 first. Near-additivity is the one that decides whether the decomposition below means anything: if the parts of a prompt do not sum to the whole, every share in every table is a share of something that was never measured. The residual is recorded rather than assumed zero, and on this run it is 0 tokens on the control text.

C1 matters for a different reason. Counting the same text twice must give the same answer, and it must do so with the cache bypassed: a count that agrees with itself because it was served from a cache has demonstrated nothing about the meter.

The models, and which weights they were

A model name is not a version. Two machines running the same tag a month apart can be running different weights, and the digest is what says whether they are.

role	model	digest
generator	qwen2.5:7b	33b70d0dad9020f1
embedder	mxbai-embed-large	81d564e4aa3a28bf

What a query costs, by configuration

Twenty-five ordinary control questions from the audit corpus, retrieved under each configuration of the published matrix, prompts built by the pipeline’s own code:

config	stack	n	prompt tokens (mean)	min..max	context share	template share	question share
A	fixed 1000, no overlap, k=3	25	488	452..537	79.2% (9659/12192)	17.2% (2100/12192)	3.7% (457/12192)
A_prime	recursive 1000, no overlap, k=3	25	482	402..509	79.0% (9525/12057)	17.4% (2100/12057)	3.8% (457/12057)
B	recursive 1000/200, k=3	25	486	426..551	79.2% (9628/12160)	17.3% (2100/12160)	3.8% (457/12160)
C	recursive 1000/200, k=8	25	1118	1040..1219	90.9% (25413/27945)	7.5% (2100/27945)	1.6% (457/27945)
D	recursive 512/128, k=8	25	760	725..820	86.7% (16466/18998)	11.1% (2100/18998)	2.4% (457/18998)
M	B + BM25 hybrid, k=8	25	1109	1022..1160	90.9% (25192/27724)	7.6% (2100/27724)	1.6% (457/27724)

Two findings worth keeping:

Widening k from 3 to 8 multiplied prompt tokens by 2.298. Scaling the context linearly with k would predict a ratio of 8/3. The measured number is 2.298x on the whole prompt and 2.64x on the retrieved context alone, because the fixed template and question dilute the ratio. The predicted ratio is arithmetic over the two configs’ recorded k, not a rule of thumb from anywhere. If a provider bills you per prompt token, k is the single biggest dial in this matrix: config C sends 1118 tokens per query where config B sends 486.

The template tax is real but proportional to nothing. The instruction scaffolding costs the same 84 tokens on every query. At k=3 with 1000-char chunks that is 17.3% of the prompt; at k=8 it falls to 7.5%. Short prompts pay the highest proportional tax, which is exactly where “add more instructions to fix the model” advice quietly compounds costs.

The tokens nobody puts in the invoice preview

Two overheads only show up when you subtract.

First, the request wrapper. The pipeline’s real generate call is non-raw, so the runtime wraps the prompt in the model’s chat template, role markers, any default system preamble and special tokens before tokenising. Measured against the raw count of the same string, that wrapper adds 29 tokens per request under config B (min 29, max 29, identical on every request in this run). We report it as one wrapper figure rather than attributing it to the chat template alone, because the subtraction cannot tell those parts apart.

Second, output boilerplate. Of 12.9 mean completion tokens per answer under config B, 323 across the 25 answers, the answer line itself is 48.6% (157/323); the citation protocol line costs 36.5% (118/323) and everything else 0.0% (0/323). The protocol is worth paying for (it feeds the deployable guardrail from the failure audit), but it should be priced as protocol, not as answer. The split is an approximate mechanical decomposition (parts tokenised separately, the residual against the true completion count is 48 tokens over 25 answers, published in the results file beside every rate).

Output tokens, every configuration

The section above takes config B apart. The rest of the matrix, for comparison:

config	n	correct	completion tokens (mean)	answer share	citation share	extra share	split residual (total)	wrapper overhead mean (min..max)
B	25	25/25	12.9	48.6% (157/323)	36.5% (118/323)	0.0% (0/323)	48	29 (29..29)
C	25	25/25	13.8	47.5% (164/345)	40.3% (139/345)	0.0% (0/345)	42	29 (29..29)

Two things to read off it. The wrapper overhead is identical on every request in every configuration, because it is the chat template and nothing in the matrix touches it. And the answer share moves very little across configurations that differ enormously in prompt size, which is the asymmetry that makes input tokens the thing to optimise: the prompt is where the configuration lives, and the completion is roughly the same sentence whatever you retrieved.

The split residual column is the same honesty as C3 above. Tokenising the parts of a completion separately and summing them does not exactly reproduce the whole, so the difference is carried in its own column rather than absorbed.

Past this embedder’s token limit, about 3037 characters of this prose, a chunk is silently unsearchable

mxbai-embed-large has a 512-token context window, read from the model itself. The embedding API truncates silently by default, and embedding token counts plateau at exactly 512 once corpus-register prose passes about 3037 characters (5.92 chars per token on this text).

The mechanism, caught directly: append a distinctive sentence to a 3340-character text and the embedding does not move at all, cosine 1.000000. The same edit at 2429 characters moves it to cosine 0.939069. The tail is stored, displayed, and returned by the vector store as if indexed. It just cannot be found.

The retrieval consequence, with head-of-document facts as the in-probe control:

config	max chunk tokens seen by embedder	tail fact retrieved	head fact retrieved (control)	severity (retrieval-only)
whole 4555-char documents as single chunks, k=3	512	0/12 (0.0%)	12/12 (100.0%)	medium
recursive 1000-char chunks, k=3	178	10/12 (83.3%)	12/12 (100.0%)	undetermined (underpowered: 2 fail / 22 pass; too few failures to grade)

Facts in document tails were retrieved 0 times out of 12 when documents were embedded whole, against 12/12 for identical-length documents with the fact at the head. Chunking under the limit recovered 10/12. Severity under the naive configuration is medium by the audit’s published criteria, assigned by the same analysis code the failure audit uses, recomputed from the recorded instances through that code as it exists now rather than assumed to be unchanged. This is retrieval-only severity: it grades whether the evidence could surface at all, not end-to-end user harm, since no generation ran here. A retrieval-score threshold does separate the failures on this corpus. The threshold is fitted on the same instances it is then scored against, with no holdout, so it is a diagnostic rather than a check the pipeline could run.

These 24 documents are near-duplicates of each other, and that is the design rather than an oversight. Every one carries the same fact sentence with the subject, register code and interval swapped: 1 template across all 24 instances, and 1 question template. Holding the sentence fixed is what makes the head side a control, because a gap between the sides then isolates position and nothing else. The bodies are alike too: filler is drawn from a pool of 10 sentences to fill each document, so any two overlap 0.87 on average and 0.90 at worst, well above the 0.8 the audit treats as near-duplicate. So read the counts above as 12 observations of one construction rather than 12 independent documents: strong evidence that the mechanism fires, and no evidence at all about how often real corpora meet it.

One scope caveat, stated so nobody overreads the hit rates: this corpus puts every identity-bearing token in the tail. A real document with a useful title or header inside the first 512 tokens may still be retrievable through that header. The claim is that the truncated portion itself is unsearchable, not a prevalence estimate for real corpora.

Note what this means for the cost tables above. The invariant is the token limit, not the character figure: the character figure is where this prose register happens to cross it under this embedder, and denser text crosses it sooner. The truncation probe is the only place this run measured a per-chunk token maximum, so whether the audited configurations truncate is not something these results state. Saying none of them do would be a claim beyond the evidence. What the run does show is that recursive 1000-char chunks, k=3 kept its largest chunk at 178 embed tokens, well inside the limit, on chunks of the size those configurations use. The configuration that gets you is the one nobody wrote down: skipping the splitter because the files seemed short.

The measurement behind the threshold

The character figure above is where this prose register crosses the token limit. It is not a property of the embedder, it is a property of this text meeting that embedder, and the sweep is what shows the difference:

input length (chars)	embed tokens	cosine(base, base + marker)	max abs vector delta	tail visible to embedder
759	134	0.933759	0.0348	yes
1518	257	0.942295	0.0367	yes
2429	416	0.939069	0.0335	yes
3340	512	1.000000	0	no
4555	512	1.000000	0	no
6074	512	1.000000	0	no

Read the last column. Below the limit, appending a distinctive sentence moves the embedding: the cosine against the unmarked text is well under one, so the tail is visible to the retriever. Above it, the cosine is exactly one and the maximum absolute change in any vector component is zero. The sentence is in the document and not in the index.

That is the failure this probe exists to demonstrate, and it is invisible from every direction a pipeline normally looks: the document loaded, the chunk embedded, the API returned success, and nothing anywhere reports that the last part of the text was discarded.

Denser text crosses the same token limit at fewer characters, which is why the heading gives a character figure for this register rather than a rule.

The models, the host, and what the digests are for

role	model	digest
generator	qwen2.5:7b	33b70d0dad9020f1
embedder	mxbai-embed-large	81d564e4aa3a28bf

The run recorded macOS-26.3-arm64-arm-64bit, Python 3.12.11, Ollama 0.15.4.

The Ollama version is load-bearing rather than decorative. Token counting here means asking the runtime for prompt_eval_count, and whether that number covers exactly the submitted text or includes a chat template is a behaviour that has changed between releases. A prompt-token figure without a runtime version beside it is a figure whose definition is unknown.

The cost model, stated plainly

This run cost zero dollars. Everything executed locally: the models, the corpus, the measurements. What transfers is the ledger, not a bill. At any provider’s published rates R_in and R_out, in dollars per million tokens:

cost per 1,000 queries = 1000 x (prompt_tokens_mean x R_in + completion_tokens_mean x R_out) / 1e6

config	prompt tokens/query (mean)	completion tokens/query (mean)	prompt tokens per 1k queries
A	488	not generated	487,700
A_prime	482	not generated	482,300
B	486	12.9	486,400
C	1118	13.8	1,117,800
D	760	not generated	759,900
M	1109	not generated	1,109,000

Multiply by your own provider’s rate sheet. We do not quote vendor prices because they change and because the token counts are the part we can stand behind.

Zero because every model call ran locally on hardware already paid for. This is not a claim that the pipeline is cheap, and it is not your cost: yours is the formula below applied to your provider’s rates. What transfers from this run is the token counts and the method, not the zero.

The zero in that ledger is the easiest number on this page to misread, which is why the run carries a sentence about it rather than only the figure. It says the hardware was already paid for. It does not say the pipeline is cheap, and it is not a discount available to anyone reading this.

Latency: what we refused to publish

The harness checked the host immediately before timing and found the second workload active, so it refused to record wall-clock and said so. The refusal record in the results file reads:

• checked at: 2026-08-30T11:55:30.068702+00:00
• reason: host contended: another workload shares the Ollama GPU
• contending processes: 3 (python3, zsh). The evidence is a count and the program names, deliberately: the raw process list names the account the machine belongs to.

That refusal is the result. A latency number taken while another loop was mid-inference on the same GPU would measure the queue, not the pipeline, and publishing it would be worse than publishing nothing.

The policy this rests on, in the run’s own words:

Token counts are recorded regardless of contention because they are deterministic given model, text and seed. Wall-clock is recorded only when host_is_contended() is False before and after every sample; otherwise the runner refuses and the refusal is the recorded result.

And the checks it rests on, one at each end of the run:

checked	contended	processes	names
at start	yes	3	python3, zsh
at end	yes	3	python3, zsh

Both ends contended, so no wall-clock figure exists anywhere in this run and none is rendered on this page. That is a worse outcome than a clean timing and a better one than a number nobody can act on: a latency figure measured against a shared GPU is a measurement of the queue, and it would transfer to nobody.

Token counts are unaffected and are published throughout, because they are deterministic given model, text and seed. The distinction is the whole reason the two are recorded under different rules.

What the truncation probe holds fixed

The probe’s instances are near-duplicates on purpose. It isolates where a fact sits inside a document, so everything except position is held still, and a denominator from it is not a count of independent observations.

property	value
documents	24
per side	12
distinct fact templates	1
distinct question templates	1
instances sharing the fact template	24
filler pool sentences	10
filler overlap, mean	0.8702
filler overlap, max	0.9043
near-duplicate threshold	0.8

Subject, register code and interval are masked out of each fact and each question, and the distinct remainders are counted: that is how many templates the instances were generated from. Document bodies are compared by token overlap, which measures the filler, the part that does vary.

A property of the instrument, rebuilt by committed code from a seed. No model call and no ground truth, so it is recomputable at any time.

Method notes

Fixed seed (7), a temperature pinned to zero in costlat/client.py rather than recorded per run, the audit’s published prompt (version v1), raw counts beside every rate, and the shared severity and detectability definitions imported from the failure audit’s analysis module, which ships inside the archive.

Be precise about what that buys, because “frozen” was doing more work than it earned. The severity classes are certified: a test recomputes them from the recorded instances through the module as it exists now, so drift between the two fails rather than hides. The corpus and stack modules are not certified that way, and both changed after this run: the config matrix has gained a configuration and some corpus documents have been reworded. So the prompt, output and cost tables are reproducible from the recorded responses, and are not claimed to be reproducible by re-running the pipeline against today’s corpus, which would measure a different one. Determinism claim: stable across replays of this run’s recorded responses on this host, not bitwise reproducibility across Ollama versions and not invariance to later changes in the shared corpus.

One part of this file is not from that run. The truncation block was re-run afterwards against the same corpus and seed, because the committed version predated fields the shared analysis module now emits. Every figure came back identical, and that is weaker corroboration than it sounds: the embedding client has no cache bypass, so the re-run replayed this run’s own recorded responses. It re-computes the current module’s arithmetic over those responses; it does not measure the models again. Every other block is the original run’s, and regenerated in the results file records which is which. Re-runs from 2026-09-05T15:07:37.355731+00:00 onward also record the Ollama build and embedder digest they checked against the original and refuse to write if either differs; earlier entries predate that check and record neither. The record holds 3 entries in total: 0 that the runner recorded as changing the block, 1 it recorded as coming back identical, and 2 from before it recorded that distinction at all, so the file does not say which those were. Per-document costs are per-query numbers times your queries per document; that multiplier belongs to your workload, not ours.