Quantization in Plain English: What Q4, Q5, Q8 Actually Cost You (July 2026)

Bottom Line (July 2026)

• What quantization is: storing each of the model’s weights as a smaller number. Fewer bits per weight means a smaller file, less memory, and faster generation, at some cost in precision.
• The ladder: FP16 is the original. Q8 is half that size and treated as indistinguishable. Q4 is a quarter of the size, noticeably faster, small real quality drop. Below Q4, quality falls off fast.
• The default: Q4_K_M. If a dynamic build exists for your model (Unsloth’s UD-Q4 variants), take that instead at roughly the same size.
• Climb only with a reason: you need spare memory and precision-sensitive work — code, math, strict JSON/tool-call output.
• Size beats precision: a bigger model at Q4 usually beats a smaller model at Q8 in the same memory budget.
• _K_M vs _K_S: medium/balanced vs small/squeezed. Pick _K_M unless the extra few hundred megabytes decide whether it fits.
• Quantizing weights does not shrink your KV cache. That is a separate memory bill — see context window traps.
• Order of operations: pick the model size your memory can hold first (what fits my machine), then pick the quant.

The Plain-English Version

A model is a giant pile of numbers called weights. When it is trained, each weight is stored at fairly high precision — usually 16 bits, written FP16 or BF16. That is why a 30B-parameter model is roughly 60GB as released: about two bytes per parameter.

Quantization compresses those numbers. Instead of 16 bits per weight, you store 8, or 5, or 4. The model gets smaller in the same way a JPEG gets smaller than a RAW photo: you throw away precision you mostly do not need, and the result still looks like the original.

The number in the name is the bits. Q8 is 8 bits per weight, Q4 is 4. Roughly:

Sizes are approximate and vary by architecture; check the actual file size on the model’s page before you plan around it.

The shape of that table is the whole lesson. Going from FP16 down to Q4 costs you very little and saves you three quarters of the memory. Going below Q4 costs you a lot and saves you comparatively little.

What the Rest of the Letters Mean

_K means K-quants — the method that has been standard in the GGUF world for a while. It does not use the same bit width everywhere; it spends more bits on the parts of the model that are most sensitive and fewer on the rest. That is why Q4_K_M holds up better than a naive flat 4-bit would.

_M vs _S vs _L are medium, small, and large variants inside the same bit level. _M is balanced, _S is squeezed a little tighter to save memory, _L spends a bit more. When you are unsure, pick _K_M. Only drop to _K_S when those few hundred megabytes decide whether the model fits.

IQ (as in IQ4_XS, IQ3_M) means the importance-matrix quants. They use a calibration pass to decide what to protect, and they hit smaller file sizes than the equivalent Q level at similar quality. The trade is that they are slightly slower to run on some backends, and at the low end they are still low-bit models with low-bit behavior.

UD (as in UD-Q4_K_XL) means the dynamic quants Unsloth publishes — the idea is to keep the layers that matter most at higher precision and compress the rest harder, so a nominally 4-bit build behaves closer to a higher one. Community reports have been strong here: one comparison in an agent harness found a UD-Q4 build matching a plain Q6 of the same model on the same tasks, with the takeaway that “Q6 spent bits for nothing.” If a UD build exists for the model you want, it is worth trying before you pay for a higher plain quant.

QAT means quantization-aware training: the model was trained with quantization in the loop rather than compressed afterward. Google has shipped Gemma QAT builds and K3-style QAT releases follow the same idea. When an official QAT build exists at 4 bits, prefer it over a community post-hoc quant of the same model.

NVFP4 / MXFP4 are the newer hardware-native 4-bit float formats, with NVFP4 accelerated on Blackwell cards. Vendor framing is usually “near-lossless,” and on chat and math benchmarks community testing has broadly agreed. But community-measured coding benchmarks have shown real regressions on some models. Treat it the way you should treat every quantization claim: it is a trade, and anyone telling you a compression is free is selling you something.

The Rules Most People Converge On

1. Q4_K_M is the default. Not because it is optimal for every case, but because it is the point where the size savings are enormous and the quality cost is small. Start here and change only when you have a reason.

2. A bigger model at Q4 usually beats a smaller model at Q8. In a fixed memory budget, parameters buy more than precision. A 30B at Q4_K_M generally outperforms a 14B at Q8. This flips only near the bottom of the ladder — a 70B at Q2 is not automatically better than a 30B at Q4.

3. Climb only with spare memory AND precision-sensitive work. The tasks where Q4 vs Q6 vs Q8 actually shows up are code generation, math, and strict structured output — anything where one wrong token breaks the result. For chat, summarizing, and drafting, the difference is hard to detect.

4. Below Q4 is a fitting compromise, not a choice. Q3 and Q2 exist so you can run a model you otherwise could not. They are worth it when the alternative is a much smaller model, and not worth it otherwise.

5. Quantizing weights does not shrink your KV cache. This is the one that catches people. Your context window lives in a separate allocation that grows with conversation length, and dropping from Q6 to Q4 does nothing to it. If your machine is fine at the start of a session and falls apart 40 messages in, that is the KV cache, and the fix is a smaller context window or KV cache quantization. Full detail in context window traps for local agents.

KV Cache Quantization Is a Different Lever

You can also quantize the KV cache itself, usually to q8 or q4. Worth knowing two things about it.

First, it is a memory lever, not a speed lever. Quantizing the cache lets you hold a longer context in the same VRAM. It does not make generation faster — f16 cache is generally faster when it fits, because there is no compression work per token.

Second, mismatched key and value cache types are a known trap. Setting the two to different types can push the backend onto a slow path and silently fall back to CPU, at which point your tokens-per-second collapses for reasons that look nothing like a cache setting. If you turn on cache quantization, set both to the same type and measure before and after. Related: why local LLMs are slow even when they fit.

Two Anecdotes Worth Keeping

One user running an agent loop kept hitting failures — malformed tool calls, steps skipped, the model claiming work it had not done. Same model, same prompts, at IQ4_XS. Moving to Q5_K_M made the failures go away. The lesson is not “IQ4 is bad”; it is that agentic work is precision-sensitive in a way chat is not, because a single malformed call breaks the whole chain. If your agent is flaky, the quant is a real suspect. See also when your local model says it did it but didn’t and local LLM tool calling reliability.

The other one is a question that gets asked in every one of these threads and almost never answered: have you actually A/B’d Q4 against Q8 on your own workload? Most people have not. They read that higher is better, paid the memory, and never measured. Run twenty of your real prompts through both and compare. It takes an hour and it settles the question for your use case permanently — which is the only place it can be settled, because “better” depends entirely on what you are asking the model to do.

The Decision Flow

1. Start with memory, not quant. Figure out what size model your machine can hold — what local LLM fits my machine walks the RAM and VRAM tiers.
2. Pick the biggest model that fits at Q4. Parameters first. If you are on a Mixture-of-Experts model, note that total parameters set the memory bill and active parameters set the speed — see llama.cpp MoE offload flags.
3. Default the quant to Q4_K_M, or a UD-Q4 build if one exists. Prefer an official QAT release at 4 bits over a post-hoc quant.
4. Reserve memory for the KV cache before you climb. Context is usually a better purchase than precision, especially for agents.
5. Climb to Q5/Q6/Q8 only after an A/B on your own workload shows a difference. If you cannot tell them apart, spend the memory on context or a bigger model.
6. Go below Q4 only to make something fit that otherwise would not.

Related Guides

• What local LLM fits my machine — size the model before you pick the quant
• Context window traps for local agents — the KV cache bill quantization does not touch
• llama.cpp MoE offload flags explained — total vs active parameters, and what to offload
• Why local LLMs are slow even when they fit — the silent CPU-fallback failure mode
• Local LLM tool calling reliability — where low quants hurt agents most
• Best local LLMs by RAM and by GPU — tier-by-tier picks
• Best local LLMs for 16GB, 24GB, 32GB, 64GB, 128GB
• Laguna XS 2.1 on 24GB vs 32GB — a worked example of quant vs context
• Usable tokens per second by task — what speed you actually need