KV Cache Quantization: q8_0 vs q4_0 vs f16 (August 2026) — What It Actually Costs You

Bottom Line (August 2026)

• It saves memory, not time. f16 is faster than q8_0 whenever it fits. Dequantization runs on every token; one measurement puts q4_0 decode at about 37% slower at 110K context.
• V-cache quantization requires flash attention. If flash attention is off or unsupported, Ollama panics on load rather than falling back gracefully. This is an open issue, not your mistake.
• The setting can silently do nothing. Quantized KV falls back to f16 on unsupported architectures without telling you. Verify with actual VRAM numbers.
• q8_0 is the safe default: roughly half the memory, quality loss most people cannot measure.
• q4_0 depends on architecture. On hybrid-attention models it has been measured as effectively lossless (BLEU 1.000 across ten configurations on Qwen 3.5). On standard full-attention models like Llama and Mistral there is real degradation.
• K is more fragile than V. If you need more than q8_0 gives you, quantize V harder before you touch K.

What the KV Cache Is, in One Paragraph

Every token you have already processed leaves behind a key and a value vector in every attention layer. That pile is the KV cache, and it is what lets the model attend to the past without recomputing it. It grows linearly with context length, it lives in VRAM alongside the weights, and at long context on a small card it can rival the model itself for size. Quantizing it means storing those vectors at 8 or 4 bits instead of 16.

That is the appeal: the KV cache is the part of your memory budget that scales with the thing you want more of.

Diagnostic Flow

Cause 1 — The Flash Attention Dependency

This is the one that produces a hard failure instead of a subtle one.

V-cache quantization in llama.cpp requires flash attention. Ollama inherits that requirement. When flash attention is auto-disabled — an unsupported backend, an architecture without support — and a quantized cache type is set anyway, the server aborts on model load rather than quietly using f16. There is an open Ollama issue arguing it should disregard the cache setting instead of panicking; until that lands, the abort is the behavior.

In llama.cpp, ask for both together explicitly:

llama-server \
  -m ./model.gguf \
  --flash-attn on \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  -c 65536 -ngl all

In Ollama, flash attention is a three-state override: unset lets Ollama decide from the backend and devices, 1 forces it on, 0 forces it off.

OLLAMA_FLASH_ATTENTION=1
OLLAMA_KV_CACHE_TYPE=q8_0

If forcing flash attention on does not resolve the panic, your model architecture genuinely lacks support and quantized KV is not available to you for that model. Unset it.

Cause 2 — It Silently Did Nothing

The quieter failure: you set the variable, the server starts fine, and nothing changed.

Quantized KV cache falls back to f16 on architectures that do not support it, without an error. Setting OLLAMA_KV_CACHE_TYPE=q8_0 is a request, not a guarantee.

There is also a much more common version of this that has nothing to do with architecture. OLLAMA_KV_CACHE_TYPE is read by the server process. Exporting it in the terminal where you type ollama run does not reach the background service. On macOS with the app running, or on Linux under systemd, you have to set it in the service environment and restart:

systemctl edit ollama
# add under [Service]:
#   Environment="OLLAMA_FLASH_ATTENTION=1"
#   Environment="OLLAMA_KV_CACHE_TYPE=q8_0"
systemctl restart ollama

Verify rather than assume. Load your model at a fixed context size and record VRAM, change the setting, restart, load again, and compare. q8_0 should visibly cut the cache portion roughly in half. If the number did not move, you did not get it — regardless of what the config says.

nvidia-smi --query-gpu=memory.used --format=csv

One more constraint worth knowing before you build around it: this is a global server setting with no per-model override. Every model that server loads gets the same cache type. If one model needs f16 fidelity and another needs the memory, you are running two servers or moving to llama.cpp, where it is a per-invocation flag.

Cause 3 — q4_0 and Your Model’s Architecture

Here is the finding that reframes the whole q4_0 question, and it is recent enough that most guides do not reflect it.

A llama.cpp issue on per-head adaptive KV quantization reports q4_0 as completely lossless on hybrid-attention models — BLEU 1.000 across ten test configurations on Qwen 3.5 at 4x compression. On standard full-attention models like Llama and Mistral, uniform q4_0 degrades quality.

The proposed explanation is that in a hybrid model only a minority of layers use full attention (the report cites 8 of 32), and the surrounding linear or sliding-attention layers act as error correction, absorbing the quantization noise. Standard architectures have no such correction because every layer is full attention, so the error accumulates instead of being damped.

The same work found that sensitivity is extremely concentrated: “sink heads,” the lowest couple of percent by entropy, are disproportionately fragile, and keeping just 3 of 144 heads unquantized outperformed spreading the same bit budget across all of them. That per-head adaptivity is a proposal, not something you can flip on today — but it explains why q4_0 feels fine to some people and clearly broken to others on nominally similar setups.

What to do with this: if you are running a hybrid-attention model, q4_0 is a much better bet than its reputation suggests. If you are on a conventional Llama- or Mistral-family model, treat q4_0 as a real trade and test it on your actual task rather than assuming the memory is free.

Cause 4 — K Is Not V

When you do need to push past q8_0, push asymmetrically.

The K cache is consistently reported as more sensitive to quantization than the V cache. The mechanism is straightforward: keys feed the dot product and softmax that decide where attention goes, so error in K changes which tokens the model looks at. Values are then averaged using those weights, and averaging tends to absorb noise rather than amplify it. Corrupting the addressing is worse than corrupting the contents.

So the sensible ladder is:

--cache-type-k f16   --cache-type-v f16    # baseline, fastest, largest
--cache-type-k q8_0  --cache-type-v q8_0   # ~half the memory, safe default
--cache-type-k q8_0  --cache-type-v q4_0   # more savings, K protected
--cache-type-k q4_0  --cache-type-v q4_0   # ~quarter memory, arch-dependent risk

The third rung is the one people skip, and it is often exactly what you need: meaningful additional savings without touching the fragile half. Ollama does not expose K and V separately — its single OLLAMA_KV_CACHE_TYPE sets both — so this particular move requires llama.cpp.

The Speed Question

Quantized KV cache does not make generation faster, and it is worth being blunt about that because “smaller means faster” is a reasonable intuition that happens to be wrong here.

Dequantization runs on every token. Reported overhead varies a lot with context length and hardware: one long-context measurement puts q4_0 decode at roughly 37% slower at 110K context, while another test at more ordinary multi-turn lengths found the f16-versus-q4_0 difference within noise, around -0.1%. The pattern across reports is that the penalty is small at moderate context and grows with cache size.

The reason to use it is that the alternatives are worse. If f16 does not fit at the context you need, your options are a shorter context window, a smaller quant of the weights, or a VRAM spill into system memory that costs you 2-3x. Against those, a modest dequantization tax is a good deal.

But if f16 fits at the context you actually run, use f16.

Sizing It Before You Commit

Cache size scales with context length, layer count, and the model’s KV head configuration. Grouped-query attention — standard on most current models — already shrinks it substantially versus older multi-head designs, which is why a 2024-era rule of thumb will overestimate badly on a 2026 model.

The reliable method is empirical, and it takes two minutes:

1. Load the model at a small context, say -c 4096, and record VRAM.
2. Load it again at your target context, say -c 65536, and record VRAM.
3. The difference is your KV cache at that context. Halve it for q8_0, quarter it for q4_0.

Now you know whether quantizing the cache actually solves your problem or whether you are a whole quant level short and should be changing the weights instead. That distinction matters, because people routinely quantize the cache to q4_0 to save a gigabyte when dropping from Q5_K_M to Q4_K_M would have saved three with less impact on quality.

Common Mistakes

1. Expecting a speed increase. It is a memory feature. f16 is faster when it fits.
2. Setting the env var in the wrong shell. Ollama reads it in the server process. Set it in the service and restart.
3. Trusting the setting instead of measuring. Silent f16 fallback is real. Compare VRAM before and after.
4. Jumping straight to q4_0. q8_0 first. It is the safe half of the savings.
5. Quantizing K and V equally when you are pushing hard. K is the fragile one. q8_0/q4_0 is a better third step than q4_0/q4_0.
6. Compressing the cache when the weights were the problem. Size the cache empirically first; sometimes the right move is a different weight quant.
7. Assuming q4_0 behaves the same across models. Architecture decides. Hybrid-attention models tolerate it far better than full-attention ones.

Related Guides

• llama.cpp MoE offload flags explained — where the KV cache fits in the wider VRAM budget
• Context window traps for local agents — why agent harnesses eat context faster than you expect
• Quantization in plain English — weight quantization, the other half of this decision
• Speculative decoding made my local LLM slower — the other optimization that commonly backfires
• Why local LLMs are slow even when they fit — the spill you are trying to avoid