How Much VRAM for 128K Context? The Exact Math (August 2026)

Bottom Line

• The KV cache formula: 2 × layers × kv_heads × head_dim × tokens × bytes_per_element.
• Qwen3-32B at 128K, FP16 cache: exactly 32 GiB. That is more than its ~19GB of Q4 weights.
• Qwen3-30B-A3B at 128K, FP16 cache: exactly 12 GiB. Nearly the same parameter count, 2.67x less cache — because it has 4 key-value heads instead of 8.
• Quantize the cache and it scales directly: q8 halves it, q4 quarters it.
• The trap: both of those models set max_position_embeddings to 40,960, not 131,072. The 128K figure requires YaRN scaling turned on deliberately.
• Rule of thumb for a dense 32B-class model: budget 0.25 GiB of FP16 KV cache per 1,000 tokens of context.

The Formula

Every transformer stores, for each token it has already seen, a key vector and a value vector at every layer. That storage is the KV cache. Its size is fully determined before you run anything:

KV bytes = 2 × layers × kv_heads × head_dim × tokens × bytes_per_element

• 2 — one key tensor and one value tensor
• layers — num_hidden_layers in config.json
• kv_heads — num_key_value_heads, not num_attention_heads
• head_dim — head_dim in config.json
• tokens — the context length you request
• bytes_per_element — 2 for FP16, 1 for q8, 0.5 for q4

The single most common error is using num_attention_heads. Modern models use grouped-query attention, where many query heads share one key-value head. Qwen3-32B has 64 attention heads but only 8 key-value heads. Use the query head count and you will overestimate the cache by 8x, conclude that long context is impossible, and buy hardware you did not need.

Worked Example 1 — Qwen3-32B (dense)

Straight from the model’s config.json:

Per token, per layer: 2 × 8 × 128 = 2,048 elements → 4,096 bytes at FP16. Across 64 layers: 262,144 bytes = 256 KiB per token.

Add roughly 19GB of Q4_K_M weights and the 128K FP16 total is about 51GB — which is why this does not fit a 24GB card, a 32GB card, or even comfortably in 48GB.

Worked Example 2 — Qwen3-30B-A3B (MoE)

Same family, similar total parameters, different attention layout:

Per token, per layer: 2 × 4 × 128 = 1,024 elements → 2,048 bytes at FP16. Across 48 layers: 98,304 bytes = 96 KiB per token.

This is the result worth internalising. Two models from the same lab, roughly the same size, and one needs 2.67x the KV cache of the other at identical context. Fewer layers (48 vs 64) and half the key-value heads (4 vs 8) compound.

Nobody markets this. Model cards advertise parameter counts and context windows; the number that decides whether long context fits on your card is num_key_value_heads, and it appears in no marketing material. If long context is your workload, read config.json before you read the benchmarks.

The 128K That Is Not 128K

Both configs above set max_position_embeddings to 40,960 tokens. Not 131,072.

The larger advertised windows come from YaRN rope scaling — a runtime configuration that extends the usable window past what the model was trained on. It is real and it works, but it is opt-in. If you request a 131,072-token context without configuring the scaling your runtime expects, one of two things happens:

1. The runtime caps you near the native limit, and your “128K context” is quietly 40K.
2. The runtime lets you exceed it, and quality degrades past the trained positions — with no error message. The model just gets worse at recalling things far back in the window.

Failure mode 2 is the dangerous one, because it looks like the model is bad rather than misconfigured. Check the model card for the exact scaling block before you size hardware around a long window. Our context window traps for local agents covers the downstream symptoms.

How to Get It to Fit

In order of how much you get per unit of effort:

1. Ask for less context. The cache is linear in tokens. Dropping a Qwen3-32B from 128K to 32K takes the FP16 cache from 32 GiB to 8 GiB. Most agent workloads never fill 128K — they fill 20K and then compact. Measure your real usage before you pay for the window.

2. Quantize the cache. q8_0 halves it and is close to free on quality for most models. q4 quarters it and is riskier. The behaviour is engine-specific and has genuine failure modes — the flash-attention dependency, the silent fallback to f16 — all covered in KV cache quantization: q8 vs q4.

3. Change model. As the two examples show, a low-KV-head architecture can cut the cache by more than half at the same capability tier. This is the biggest lever and the least discussed.

4. Then buy memory. Only after the first three. Best hardware for 1M context covers the tier where you genuinely cannot optimise your way out.

Quick Sizing Table

FP16 KV cache at 128K context, by architecture shape:

Find your model’s three numbers in config.json, match the row, halve for q8, quarter for q4.

Common Mistakes

• Using num_attention_heads. Overestimates by the grouped-query ratio, often 8x.
• Assuming the cache scales with parameter count. It scales with layers and KV heads. A 30B and a 32B can differ by 2.67x.
• Forgetting the cache is per-sequence. Serving four concurrent requests at 32K each costs the same as one at 128K.
• Sizing for weights only. At long context the cache is the larger allocation, not the smaller one.
• Trusting an advertised context window without checking max_position_embeddings.

See Also

• KV Cache Quantization: q8_0 vs q4_0 vs f16 — how to actually halve or quarter these numbers
• Context Window Traps for Local Agents — what goes wrong when the window is misconfigured
• Best Hardware for 1M Context Locally — the tier above, where the cache dominates everything
• 128GB Local LLM Context Window — what a 128GB machine buys you in context terms
• Quantization in Plain English — the weight side of the same memory budget
• MoE vs Dense on 24GB — why architecture beats parameter count at this tier
• Is 48GB of VRAM Enough in 2026? — this formula applied to a dense 70B, whose 128K cache is exactly 40 GiB

Sources

• config.json for Qwen/Qwen3-32B — layers, attention heads, key-value heads, head dimension, max_position_embeddings (40,960)
• config.json for Qwen/Qwen3-30B-A3B — layers, key-value heads, head dimension, expert counts, max_position_embeddings (40,960)
• All cache sizes above are computed from the standard KV cache formula applied to those published fields. They are arithmetic, not measurements — real allocations add small per-engine overhead.