Which local model uses the least KV cache for long context?

Of the models in this table, gpt-oss 20B is the cheapest at 3.0 GiB for a 131,072-token window, followed by gpt-oss 120B at 4.5 GiB. Both are unusually cheap because half their layers use a 128-token sliding window and can never grow, and because they use a head dimension of 64 where most current models use 128. Among models with no sliding window, Qwen3-30B-A3B is the cheapest at 12.0 GiB, on 4 key-value heads across 48 layers.

What determines how much KV cache a model uses?

Three config fields and nothing else: num_hidden_layers, num_key_value_heads and head_dim. The cache holds one key and one value per key-value head, per layer, per token, so the per-token cost at fp16 is 2 times layers times kv_heads times head_dim times 2 bytes. Total parameter count does not appear in that formula. Neither does the expert count of a mixture-of-experts model. That is why cache cost and model size rank so differently.

Why does a 21B model use less KV cache than an 8B model?

Because the two were designed with different attention shapes. gpt-oss 20B has 24 layers of which only 12 grow with context, 8 key-value heads and a head dimension of 64, giving 24 KiB per token. Qwen3-8B has 36 layers that all grow, 8 key-value heads and a head dimension of 128, giving 144 KiB per token. That is six times more cache for a model with roughly a third of the parameters. Parameter count tells you about weights; it tells you nothing about context cost.

Does a mixture-of-experts model use less KV cache?

Often yes, but not because it is a mixture of experts. The routing affects the feed-forward layers, not attention, so it never appears in the cache formula. What tends to help is that MoE designs frequently use fewer key-value heads. Qwen3-30B-A3B and Qwen3-235B-A22B both use 4 key-value heads where the dense Qwen3 models use 8, which halves their cache per layer. Qwen3-235B-A22B costs 23.5 GiB per 128K window against Qwen3-32B's 32.0 GiB, despite being over seven times the parameters.

Are these KV cache numbers measured?

No. Every figure here is arithmetic derived from published config.json fields, read on 6 September 2026. Real allocations differ. Runtimes add compute buffers, some pad the cache, and a runtime that ignores a sliding window will reserve double for the gpt-oss models. Cache quantisation to 8-bit halves every figure and to 4-bit quarters it. Use this table to rank models against each other, then verify the allocation your own loader reports.

← All guides

Models September 6, 2026

Lowest KV Cache Local LLM (2026): gpt-oss 20B Wins

Model cards publish parameter counts and benchmark scores. None of them publish the number that decides whether a long context fits your card: the KV cache cost per token. It is not proportional to model size. A 21B model can cost six times less cache than an 8B one, and a 235B mixture-of-experts can cost less than a 32B dense model. This page ranks nine current local models by cache per 128K context, computed from fields in their own config files.

Bottom Line

KV cache cost does not track model size. It tracks three config fields: layers, key-value heads and head dimension.
gpt-oss 20B is the cheapest here at 3.0 GiB per 131,072-token window. Llama 3.3 70B is the most expensive at 40.0 GiB — a 13x spread.
A 21B model costs six times less cache than an 8B one. gpt-oss 20B is 3.0 GiB; Qwen3-8B is 18.0 GiB.
A 235B model costs less than a 32B one. Qwen3-235B-A22B is 23.5 GiB; Qwen3-32B is 32.0 GiB.
num_key_value_heads is the field that decides it, and it appears on no model card headline, no benchmark chart and no product page.
Halving key-value heads halves the cache exactly. There is no approximation in that.
These are calculations, not measurements. Your runtime will report something else. Rank with this table; size with your loader.

The Table

Every field comes from each model’s published config.json on Hugging Face, read on 6 September 2026. The cache figures are fp16, unquantised.

Model	Layers that grow	KV heads	`head_dim`	KiB / token	GiB @ 32,768	GiB @ 131,072
gpt-oss 20B	12 of 24	8	64	24	0.75	3.0
gpt-oss 120B	18 of 36	8	64	36	1.1	4.5
Qwen3-30B-A3B	48 of 48	4	128	96	3.0	12.0
Qwen3-8B	36 of 36	8	128	144	4.5	18.0
Qwen3-14B	40 of 40	8	128	160	5.0	20.0
Mistral Small 3.2 24B	40 of 40	8	128	160	5.0	20.0
Qwen3-235B-A22B	94 of 94	4	128	188	5.9	23.5
Qwen3-32B	64 of 64	8	128	256	8.0	32.0
Llama 3.3 70B	80 of 80	8	128	320	10.0	40.0

Sort that table by parameter count and it scrambles. The ordering by cache and the ordering by size are close to unrelated.

The Formula

The cache holds one key and one value for each key-value head, in every layer, for every token. At fp16 that is two bytes per value:

bytes per token = 2 (K and V) x layers x kv_heads x head_dim x 2 bytes

Worked for Qwen3-32B:

2 x 64 x 8 x 128 x 2 = 262,144 bytes = 256 KiB per token
256 KiB x 131,072 tokens = 32 GiB

Worked for gpt-oss 20B, where only the 12 full-attention layers count:

2 x 12 x 8 x 64 x 2 = 24,576 bytes = 24 KiB per token
24 KiB x 131,072 tokens = 3.0 GiB

Notice what is absent from that formula. Total parameters. Expert count. Quantisation of the weights. Benchmark scores. None of them touch the cache. This is why a page that tells you “a 70B needs 40GB at Q4” has told you about half the problem — the other 40 GiB is context, and it is a different calculation entirely. Our 128K context VRAM page works the general version.

The Two Design Choices That Matter

Key-value heads

This is the strongest lever and the least visible one. Grouped-query attention lets a model use fewer key-value heads than query heads, and the cache scales directly with that count.

Qwen3-32B and Qwen3-30B-A3B are instructive because they are close in name and far apart in cache. The dense 32B uses 8 key-value heads across 64 layers. The MoE 30B uses 4 across 48 layers. The result is 32.0 GiB against 12.0 GiB — a 2.7x difference between two models a buyer would treat as interchangeable.

Sliding-window attention

The gpt-oss pair go further and simply stop half their layers from growing. Both alternate sliding_attention and full_attention, with sliding_window set to 128. Twelve of the 20B’s layers, and eighteen of the 120B’s, hold 128 tokens each forever. That is about 3.0 MiB in total for the 20B — a rounding error against the 3.0 GiB the other half costs.

This is not free. In those layers a token can attend no further back than 127 positions. Long-range recall is removed by construction, and the model compensates through its full-attention layers. If your work depends on precise recall of something far back in the window, test that specifically. We cover the tradeoff on the gpt-oss 120B sizing page.

The Caveat On Window Length

The right column compares every model at 131,072 tokens so the arithmetic is comparable. Not every model here has a 131,072-token native window.

Native `max_position_embeddings`	Models
131,072	gpt-oss 20B, gpt-oss 120B, Mistral Small 3.2 24B, Llama 3.3 70B
40,960	Qwen3-8B, Qwen3-14B, Qwen3-32B, Qwen3-30B-A3B, Qwen3-235B-A22B

The Qwen3 models reach 128K through RoPE scaling, not natively. Enabling it changes quality as well as memory, so the 32,768-token column is the fairer comparison if you are choosing between a Qwen3 model and a gpt-oss one — every model in the table runs that window without scaling.

How To Use This

The practical move is the one people skip: change the model before you buy the card.

If you have 16GB and a 24GB cache requirement, a second card costs more than every model in this table combined. Switching from Qwen3-32B to Qwen3-30B-A3B saves 20 GiB of cache at 128K for no money. Switching to gpt-oss 20B saves 29 GiB.

If you have 24GB, the whole gpt-oss 20B job — weights plus a full 128K cache — fits with headroom, while Qwen3-32B’s cache alone overflows the card before its weights are loaded. That is the decision the 24GB tier page is really about.

If you have 96GB or more, the cache stops being the constraint and the weights take over. See is 96GB of VRAM enough.

And if none of that is enough, KV cache quantisation halves every figure in the table at 8-bit. Do that after you have picked the cheapest model that does your work, not before. Quantising a bad choice is more expensive in quality than choosing well was in memory.

Sources

Read 2026-09-06. Each link is the raw config.json for the named model.

openai/gpt-oss-20b — 24 layers, alternating layer types, sliding_window 128, 8 KV heads, head_dim 64
openai/gpt-oss-120b — 36 layers, alternating layer types, sliding_window 128, 8 KV heads, head_dim 64
Qwen/Qwen3-8B — 36 layers, 8 KV heads, head_dim 128, max_position_embeddings 40,960
Qwen/Qwen3-14B — 40 layers, 8 KV heads, head_dim 128
Qwen/Qwen3-32B — 64 layers, 8 KV heads, head_dim 128
Qwen/Qwen3-30B-A3B — 48 layers, 4 KV heads, head_dim 128, 128 experts, 8 per token
Qwen/Qwen3-235B-A22B — 94 layers, 4 KV heads, head_dim 128
mistralai/Mistral-Small-3.2-24B-Instruct-2506 — 40 layers, 8 KV heads, head_dim 128, max_position_embeddings 131,072
Llama-3.3-70B-Instruct config — 80 layers, 8 KV heads, hidden_size 8192 across 64 heads gives head_dim 128. Read from the open mirror because Meta’s own repository is access-gated.

These are calculations, not measurements. No benchmark was run for this page and none is implied. Runtimes add buffers, pad allocations, and in the gpt-oss case may ignore the sliding window and reserve double. Check what your loader reports against the row above before you conclude a model does or does not fit.

Need OpenClaw fixed live?

Remote rescue sessions for gateway, auth, tunnel, VPS, and model access problems.

See Rescue Session