Local Model Calculator

Does the model fit at your context length? File size is half the answer. The KV cache grows with every token, so a model that loads at 4K can fail at 32K. Pick your hardware, a model, a context and the cache type you run (f16, q8_0 or q4_0).

Models and their cache cost

Layer counts, KV heads and head size come from each model's config.json on Hugging Face. File sizes come from the GGUF repository's file list. Both were read on 2026-11-13.

ModelQ4-class fileKV cache per token (f16)Cache at 32K128K, f16128K, q8_0128K, q4_0Max context
Qwen3.5 9B Q4_K_M, 5.29 GiB 32 KiB (8 of 32 layers) 1.00 GiB 4.00 GiB 2.13 GiB 1.13 GiB 256K
gpt-oss 20B MXFP4, 11.28 GiB 24 KiB (12 of 24 layers) 0.77 GiB 3.02 GiB 1.60 GiB 0.85 GiB 128K
Qwen3.8 27B UD-Q4_K_M, 15.43 GiB 64 KiB (16 of 64 layers) 2.00 GiB 8.00 GiB 4.25 GiB 2.25 GiB 256K
Qwen3.6 35B-A3B UD-Q4_K_M, 20.61 GiB 20 KiB (10 of 40 layers) 0.63 GiB 2.50 GiB 1.33 GiB 0.70 GiB 256K
gpt-oss 120B MXFP4, 59.03 GiB 36 KiB (18 of 36 layers) 1.15 GiB 4.53 GiB 2.40 GiB 1.27 GiB 128K
Ornith 1.5 122B UD-Q4_K_XL, 74.24 GiB 24 KiB (12 of 48 layers) 0.75 GiB 3.00 GiB 1.59 GiB 0.84 GiB 256K

How the number is calculated

memory needed = file size + KV cache
KV cache      = context tokens x full-attention layers x 2 (K and V) x KV heads x head size x bytes per value
bytes/value   = f16: 2    q8_0: 34 / 32 = 1.0625    q4_0: 18 / 32 = 0.5625

Hardware for each memory tier

Prices as of November 2026. They move weekly in 2026, so check the listing. Rental rates are RunPod on-demand, Secure Cloud and Community Cloud, from runpod.io/pricing (page updated November 12, 2026).

MemoryPickPriceRent the same memory
12 GB RTX 3060 12GB Check current listings Not on the list
16 GB RTX 5060 Ti 16GB $800 lowest US retail price (GPUPrix, Newegg/Best Buy/B&H, November 4, 2026) Not on the list
24 GB RTX 3090 24GB (Amazon Renewed) $1,626 used average on 313 listings (resaleprices.com, November 13, 2026) RTX 3090 24GB: $0.50 / $0.22 per hour
32 GB RTX 5090 32GB $6,331 to $6,643 lowest daily average (GPU Poet, November 7-8, 2026) RTX 5090 32GB: $1.39 / $0.69 per hour
48 GB RTX A6000 48GB Check current listings A40 48GB: $0.59 / $0.35 per hour
96 GB RTX PRO 6000 Blackwell 96GB $15,299.99 at Central Computer (November 9, 2026) RTX PRO 6000 96GB: $2.69 / $1.69 per hour
128 GB ASUS Ascent GX10 128GB (same GB10 chip, open box on Amazon) Check current listings Not on the list

Affiliate links. The full list by budget is on the gear page.

Questions

How much VRAM does context length use?

Context is stored in the KV cache, and its size depends on the model. Qwen3.8 27B has 16 full-attention layers, 4 KV heads and a head size of 256, which is 64 KiB per token at f16. That is 2.00 GiB at 32K, 8.00 GiB at 128K and 16.00 GiB at 256K, on top of the weights. Qwen3.6 35B-A3B needs 20 KiB per token and gpt-oss 20B needs 24 KiB.

Does Qwen3.8 27B fit on a 24GB GPU?

Yes at short and medium context. The UD-Q4_K_M file is 15.43 GiB. With 32K of context it needs 17.43 GiB, and with 64K it needs 19.43 GiB. Both fit a 24GB card with 1 GiB kept free. At 128K it needs 23.43 GiB, which does not fit. Use a smaller file or a shorter context.

Why does a model load at 4K context and fail at 32K or 128K?

The weights stay the same size, but the KV cache grows with every token of context. A file that leaves 2 GiB free at 4K has no room left once the cache passes 2 GiB. For Qwen3.8 27B the cache reaches 2 GiB at 32K. This calculator adds the cache to the file size before it gives a verdict.

How much VRAM does a q8_0 KV cache save?

A q8_0 block stores 32 values in 34 bytes, so the cache takes 53.1% of the f16 size, not exactly half. q4_0 stores 32 values in 18 bytes, 28.1% of f16. For Qwen3.8 27B at 128K that is 8.00 GiB at f16, 4.25 GiB at q8_0 and 2.25 GiB at q4_0. On a 24GB card the saving buys a bigger file: the largest one that fits at 128K is UD-IQ4_XS with an f16 cache, UD-Q5_K_M with q8_0 and UD-Q6_K with q4_0.

How do I turn on a q8_0 KV cache?

In llama.cpp, add -ctk q8_0 -ctv q8_0. A quantized V cache needs flash attention. With the default -fa auto, llama.cpp turns flash attention on for you; with -fa off it refuses to start. In Ollama, set OLLAMA_KV_CACHE_TYPE=q8_0 on the server and restart it. The setting applies to every model, and it works only when flash attention is on.

How much memory does a 256K context need?

For Qwen3.8 27B, 256K of context is 16.00 GiB of KV cache at f16, so the UD-Q4_K_M file plus cache needs 31.43 GiB. For Qwen3.6 35B-A3B the same context is 5.00 GiB of cache. gpt-oss 20B stops at 128K, the maximum in its config.

Submit your benchmark

Ran one of these models? Send the hardware, model, quantization and tokens per second. Good submissions become benchmark notes.

Related guides

Generate DITA/XML docs Turn setup notes into structured documentation on MarkdownMe. Practice AI cert questions Use CloudYeti's free AWS AI Practitioner diagnostic.