Does the model fit at your context length? File size is half the answer. The KV cache grows with every token, so a model that loads at 4K can fail at 32K. Pick your hardware, a model, a context and the cache type you run (f16, q8_0 or q4_0).
Layer counts, KV heads and head size come from each model's config.json on Hugging Face. File sizes come from the GGUF repository's file list. Both were read on 2026-11-13.
| Model | Q4-class file | KV cache per token (f16) | Cache at 32K | 128K, f16 | 128K, q8_0 | 128K, q4_0 | Max context |
|---|---|---|---|---|---|---|---|
| Qwen3.5 9B | Q4_K_M, 5.29 GiB | 32 KiB (8 of 32 layers) | 1.00 GiB | 4.00 GiB | 2.13 GiB | 1.13 GiB | 256K |
| gpt-oss 20B | MXFP4, 11.28 GiB | 24 KiB (12 of 24 layers) | 0.77 GiB | 3.02 GiB | 1.60 GiB | 0.85 GiB | 128K |
| Qwen3.8 27B | UD-Q4_K_M, 15.43 GiB | 64 KiB (16 of 64 layers) | 2.00 GiB | 8.00 GiB | 4.25 GiB | 2.25 GiB | 256K |
| Qwen3.6 35B-A3B | UD-Q4_K_M, 20.61 GiB | 20 KiB (10 of 40 layers) | 0.63 GiB | 2.50 GiB | 1.33 GiB | 0.70 GiB | 256K |
| gpt-oss 120B | MXFP4, 59.03 GiB | 36 KiB (18 of 36 layers) | 1.15 GiB | 4.53 GiB | 2.40 GiB | 1.27 GiB | 128K |
| Ornith 1.5 122B | UD-Q4_K_XL, 74.24 GiB | 24 KiB (12 of 48 layers) | 0.75 GiB | 3.00 GiB | 1.59 GiB | 0.84 GiB | 256K |
memory needed = file size + KV cache KV cache = context tokens x full-attention layers x 2 (K and V) x KV heads x head size x bytes per value bytes/value = f16: 2 q8_0: 34 / 32 = 1.0625 q4_0: 18 / 32 = 0.5625
src/models/qwen35.cpp, read 2026-11-13).src/llama-kv-cache-iswa.cpp, commit 7a2e5aba90).ggml/src/ggml-common.h, llama.cpp master e574a17b06, read 2026-11-14), so the cache is 53.1% or 28.1% of f16. The tool uses one type for K and V. llama.cpp sets them apart with -ctk and -ctv.-fa is auto (the default) and refuses to load with -fa off (src/llama-context.cpp). Ollama applies OLLAMA_KV_CACHE_TYPE only when flash attention is on. Ollama's FAQ calls q8_0 "usually no noticeable impact" and q4_0 a "small-medium loss in precision that may be more noticeable at higher context sizes" (docs/faq, read 2026-11-14). The trade-offs are in KV cache quantization: start with q8_0.Prices as of November 2026. They move weekly in 2026, so check the listing. Rental rates are RunPod on-demand, Secure Cloud and Community Cloud, from runpod.io/pricing (page updated November 12, 2026).
| Memory | Pick | Price | Rent the same memory |
|---|---|---|---|
| 12 GB | RTX 3060 12GB | Check current listings | Not on the list |
| 16 GB | RTX 5060 Ti 16GB | $800 lowest US retail price (GPUPrix, Newegg/Best Buy/B&H, November 4, 2026) | Not on the list |
| 24 GB | RTX 3090 24GB (Amazon Renewed) | $1,626 used average on 313 listings (resaleprices.com, November 13, 2026) | RTX 3090 24GB: $0.50 / $0.22 per hour |
| 32 GB | RTX 5090 32GB | $6,331 to $6,643 lowest daily average (GPU Poet, November 7-8, 2026) | RTX 5090 32GB: $1.39 / $0.69 per hour |
| 48 GB | RTX A6000 48GB | Check current listings | A40 48GB: $0.59 / $0.35 per hour |
| 96 GB | RTX PRO 6000 Blackwell 96GB | $15,299.99 at Central Computer (November 9, 2026) | RTX PRO 6000 96GB: $2.69 / $1.69 per hour |
| 128 GB | ASUS Ascent GX10 128GB (same GB10 chip, open box on Amazon) | Check current listings | Not on the list |
Affiliate links. The full list by budget is on the gear page.
Context is stored in the KV cache, and its size depends on the model. Qwen3.8 27B has 16 full-attention layers, 4 KV heads and a head size of 256, which is 64 KiB per token at f16. That is 2.00 GiB at 32K, 8.00 GiB at 128K and 16.00 GiB at 256K, on top of the weights. Qwen3.6 35B-A3B needs 20 KiB per token and gpt-oss 20B needs 24 KiB.
Yes at short and medium context. The UD-Q4_K_M file is 15.43 GiB. With 32K of context it needs 17.43 GiB, and with 64K it needs 19.43 GiB. Both fit a 24GB card with 1 GiB kept free. At 128K it needs 23.43 GiB, which does not fit. Use a smaller file or a shorter context.
The weights stay the same size, but the KV cache grows with every token of context. A file that leaves 2 GiB free at 4K has no room left once the cache passes 2 GiB. For Qwen3.8 27B the cache reaches 2 GiB at 32K. This calculator adds the cache to the file size before it gives a verdict.
A q8_0 block stores 32 values in 34 bytes, so the cache takes 53.1% of the f16 size, not exactly half. q4_0 stores 32 values in 18 bytes, 28.1% of f16. For Qwen3.8 27B at 128K that is 8.00 GiB at f16, 4.25 GiB at q8_0 and 2.25 GiB at q4_0. On a 24GB card the saving buys a bigger file: the largest one that fits at 128K is UD-IQ4_XS with an f16 cache, UD-Q5_K_M with q8_0 and UD-Q6_K with q4_0.
In llama.cpp, add -ctk q8_0 -ctv q8_0. A quantized V cache needs flash attention. With the default -fa auto, llama.cpp turns flash attention on for you; with -fa off it refuses to start. In Ollama, set OLLAMA_KV_CACHE_TYPE=q8_0 on the server and restart it. The setting applies to every model, and it works only when flash attention is on.
For Qwen3.8 27B, 256K of context is 16.00 GiB of KV cache at f16, so the UD-Q4_K_M file plus cache needs 31.43 GiB. For Qwen3.6 35B-A3B the same context is 5.00 GiB of cache. gpt-oss 20B stops at 128K, the maximum in its config.
Ran one of these models? Send the hardware, model, quantization and tokens per second. Good submissions become benchmark notes.