Hardware October 14, 2026

Best Local LLM for 8GB VRAM (2026): Qwen3.5 9B Wins

On an 8GB card, run Qwen3.5 9B at Q4_K_M. The 5.29 GiB file leaves room for 32K of context at full precision. Every 20B and 27B file is too big to run fully on the card with useful context. If you have 32GB of system RAM, Qwen3.6 35B-A3B with its experts in RAM is the stronger model at about 30 tok/s.

Running an agent on an 8GB card?

Use the local model calculator first, then see our AI training options if you want help sizing the context window so the card does not spill into system RAM.

Bottom Line

Best LLM for 8GB VRAM: Qwen3.5 9B at Q4_K_M. Unsloth’s file is 5.29 GiB. It leaves room for 32K of context at f16 or 64K with a Q8 cache.
Best Qwen model for 8GB VRAM: the same Qwen3.5 9B. There is no 9B in Qwen3.6 or Qwen3.8. For agents, use Xiaomi’s fine-tune MiMo-V2.6-Distill-Qwen-9B at Q4_K_M (5.44 GiB).
No 20B or 27B file runs fully on the card with useful context. gpt-oss 20B is 11.28 GiB. Underdog Saluki 27B is a 7.36 GiB file, and with every layer on the card it leaves no room for a KV cache.
Stronger, if you have 32GB of system RAM: Qwen3.6 35B-A3B with its experts in RAM. One tester measured 31.8 tok/s using 6.1 GB of VRAM (community-reported).
Do not buy a new 8GB card for local AI. As of October 1, 2026, a new Arc B580 12GB is $329.99 and a used RTX 3060 12GB averages $307.

Ready to buy? See the tested hardware list with current prices.

How Much of an 8GB Card You Can Use

An “8GB” card has 8 GiB of memory: 8,192 MiB. The model cannot use all of it.

Driver and CUDA context: one RTX 3060 owner reports 378 MB reserved by the Linux driver on a headless card (llama.cpp discussion #15396, community-reported). We assume about 0.4 GiB.
Compute buffers: llama.cpp allocates scratch memory for each batch. We assume about 1 GiB at default batch sizes. This is our assumption, not a measurement. A smaller -ub lowers it.
Your desktop: if a monitor runs on the same card, Windows or your compositor takes more. Check nvidia-smi before you load a model and subtract what it shows.

That leaves about 6.6 GiB for weights plus KV cache on a headless card. Every “fits” verdict on this page uses that budget. On a card that also drives your desktop, plan for about 6 GiB.

What Fits in 8GB VRAM

Every file size is a byte count from the Hugging Face API, read 2026-10-14. GiB = 2³⁰ bytes. Totals include weights, the fixed linear-attention state and the KV cache.

Model	File	Size (GiB)	Context in ~6.6 GiB	Verdict
Qwen3.5 9B	Q4_K_M (Unsloth)	5.29	32K f16 (6.34 total), or 64K Q8 (6.40)	Best pick
MiMo-V2.6-Distill-Qwen-9B	Q4_K_M (bartowski)	5.44	32K f16 (6.49 total)	Agent and coding swap
Qwen3.5 9B	Q5_K_M (Unsloth)	6.13	16K Q8 (6.44 total)	Higher quality, short context
Qwen3.5 9B	Q6_K (Unsloth)	6.95	None	Over budget
Gemma 4 E4B	QAT Q4_0 (Google)	4.80	1.8 GiB free for cache	Fits easily; not sized here
Gemma 4 12B	QAT Q4_0 (Google)	6.50	About 0.1 GiB	No useful context
Ternary Bonsai 2 27B	PTQ1_0 (Prism ML)	5.54	About 28K Q8 (fork only)	Fits, needs a fork
Underdog Saluki 27B	IQ2-mix (ConwayResearch)	7.36	None at `-ngl 99`	Does not fit with real context
gpt-oss 20B	MXFP4 (ggml-org)	11.28	None on GPU	Use `--n-cpu-moe`
Qwen3.6 35B-A3B	UD-Q4_K_M (Unsloth)	20.61	Experts in system RAM	31.8 tok/s at 6.1 GB VRAM (community)

Hardware line: the common 8GB cards are the RTX 4060, RTX 5060, RTX 4060 Ti 8GB, RTX 5060 Ti 8GB, RTX 3070, RTX 3060 Ti, RTX 2080 and AMD RX 7600. They all hold the same files, because they all hold 8 GiB. They differ in speed.

Why Qwen3.5 9B wins

It is the strongest model that fits fully on the card with real context. Qwen’s model card scores Qwen3.5 9B at 65.6 on LiveCodeBench v6, against 74.6 for gpt-oss 20B in the same table. gpt-oss 20B is stronger, but it does not fit on 8GB (see below).

Its cache is small. The config.json lists 32 layers with full attention on every 4th layer, 4 KV heads and a head dimension of 256. Only 8 layers keep a cache that grows.

2 (K and V) x 8 layers x 4 KV heads x 256 head_dim x 2 bytes = 32 KiB per token (f16)
Q8 cache: about 17 KiB per token
Fixed linear-attention state: 24 layers x 32 heads x 128 x 128 x 4 bytes = 48 MiB, plus 2 MiB conv state

Context	Q4_K_M, f16 cache	Q4_K_M, Q8 cache	Q5_K_M, Q8 cache
16K	5.84 GiB	5.61 GiB	6.44 GiB
32K	6.34 GiB, fits	5.87 GiB	6.71 GiB, just over
64K	7.34 GiB, no	6.40 GiB, fits	7.24 GiB, no

This is our arithmetic from the published config. Compare each total with the 6.6 GiB budget above. The largest f16 context Q4_K_M holds in that budget is about 41K tokens.

Skip the vision projector. Unsloth’s mmproj-F16 file adds 0.86 GiB. On 8GB that costs about 27K tokens of f16 context.

The agent swap: MiMo-V2.6-Distill-Qwen-9B

Xiaomi published this fine-tune of Qwen3.5 9B on 2026-09-21. Its model card scores it 37.1 on Terminal Bench 2.1 against 27.0 for the base 9B, and 44.6 against 32.0 on SWE Pro. These are vendor numbers from Xiaomi’s technical report. The architecture is the same, so the cache math above applies. bartowski’s Q4_K_M is 5.44 GiB, 0.15 GiB more than Unsloth’s Qwen3.5 file, and still fits 32K of f16 context.

How to run it

llama-server -hf unsloth/Qwen3.5-9B-GGUF:Q4_K_M \
  --jinja -ngl 99 -fa on -c 32768

For 64K, add --cache-type-k q8_0 --cache-type-v q8_0 and set -c 65536. If a monitor runs on the card and the model fails to load, drop -c to 16384.

Speed by 8GB Card

Generation speed scales roughly with the bytes the card reads per token. The table below starts from the llama.cpp scoreboards, which test a 3.56 GiB Llama 2 7B Q4_0 file. We scale each result to the 5.29 GiB Qwen3.5 9B file.

Card	Memory (NVIDIA spec)	Llama 7B Q4_0, tok/s (measured)	Qwen3.5 9B Q4_K_M ceiling (our estimate)
RTX 5060 8GB	GDDR7, 128-bit	100.61 (CUDA, FA on)	About 68
RTX 3060 Ti 8GB	not checked	96.50 (CUDA, FA on)	About 65
RTX 2070 Super 8GB	not checked	87.71 (CUDA, FA on)	About 59
RTX 3070 8GB	not checked	78.71 (Vulkan, FA off)	About 53
RTX 4060 Ti 8GB	GDDR6, 128-bit	64.03 (CUDA, FA on)	About 43
AMD RX 7600 8GB	not checked	60.88 (Vulkan, FA on)	About 41
RTX 4060 Laptop 8GB	not checked	59.15 (CUDA, FA on)	About 40
RTX 3060 12GB, for comparison	not checked	76.92 (CUDA, FA on)	About 52

Sources: the CUDA scoreboard table and comments, and the Vulkan scoreboard. Each row is one community run. Qwen3.5 mixes linear and full attention, and its kernels can be slower than plain Q4_0. Read the right column as a ceiling.

The RTX 4060 Ti 8GB generates slower than a 3060 12GB. It scores 64.03 against 76.92 on the same CUDA scoreboard, and it holds 4 GiB less. NVIDIA lists a 128-bit memory bus for the whole RTX 4060 and 5060 family. The RTX 5060 gets its speed from GDDR7, not from a wider bus.

Why 20B and 27B Models Do Not Fit

gpt-oss 20B: use expert offload

gpt-oss 20B is our 16GB pick. On 8GB the file is larger than the card.

Card:                        8,192 MiB
MXFP4 weights:             -11,548.6 MiB   (12,109,566,624 bytes)
Short by:                   3,356.6 MiB before any context

Smaller quants do not help. OpenAI trained the expert weights in 4-bit MXFP4. Unsloth’s smallest file, Q2_K, is still 10.68 GiB (11,468,317,888 bytes). Any claim that gpt-oss 20B fits in 6 GB at 2 bits does not match the published files.

The honest route is MoE offload. llama.cpp’s --n-cpu-moe N keeps the expert weights of N layers in system RAM. Attention and the KV cache stay on the GPU. The official llama.cpp gpt-oss guide gives this example for an RTX 2060 8GB:

# 32K context, 16 of 24 MoE layers on the CPU
llama-server -hf ggml-org/gpt-oss-20b-GGUF --ctx-size 32768 \
  --jinja -ub 2048 -b 2048 --n-cpu-moe 16

One community user in the same discussion measured 42.32 tok/s generation and about 1,640 tok/s prompt processing on an RTX 4060 Laptop 8GB with --n-cpu-moe 14, a Core i7-14700HX and 64GB of DDR5-5200. The same laptop ran gpt-oss 120B at 18.85 tok/s with --n-cpu-moe 34. Both are community-reported. Full flag guide: llama.cpp MoE offload flags explained.

Qwen3.6 35B-A3B: the strongest model an 8GB card can drive

Qwen’s model card scores Qwen3.6 35B-A3B at 80.4 on LiveCodeBench v6. That is well above the 9B’s 65.6, though the two scores come from separate Qwen tables. Only about 3B parameters fire per token, so the experts can live in system RAM.

InsiderLLM swept --n-cpu-moe on an RTX 3060 12GB with 32GB of DDR4 and Unsloth’s 20.61 GiB UD-Q4_K_M file (source, firsthand, community-reported):

`--n-cpu-moe`	Expert layers on GPU	VRAM used	tok/s (empty context)	Fits 8GB?
40 (all)	0 of 40	2.5 GB	28.1	Yes
32	8 of 40	6.1 GB	31.8	Yes, headless
24	16 of 40	9.8 GB	38.9	No

The test card has 12GB, but the first two rows use less than 8GB. Our inference: an 8GB card with similar system RAM runs -ncmoe 32 at about the same speed, because system RAM sets the pace. If a monitor runs on the card, start at 34 or 36. Its cache is 20 KiB per token (10 full-attention layers x 2 x 2 KV heads x 256 x 2 bytes), so context is cheap. You need at least 32GB of system RAM.

Underdog Saluki 27B: under 8 GB, but not on an 8GB card

ConwayResearch’s model card says “Qwen3.8-27B in under 8 GB”. The file is 7,898,369,152 bytes, or 7,532.5 MiB. llama.cpp keeps the input embedding (265 MiB in this file) in CPU RAM, so about 7,257 MiB goes to the card. The card’s own quickstart runs it with -c 32768. Here is the arithmetic on an 8GB card with one slot (-np 1), using the Qwen3.8 27B config.json (16 full-attention layers, 4 KV heads, head_dim 256):

Card:                                8,192 MiB
Weights on the GPU:                 -7,257 MiB
Linear-attention + conv state:        -150 MiB
Compute buffer (llama.cpp #29063):  -516 to -839 MiB
Left for KV cache AND driver:       +269 to -54 MiB

KV cache: 64 KiB per token at f16
32K at f16 needs 2.0 GiB; at Q8 about 1.06 GiB

The CUDA context also needs memory (one owner reports 378 MB). So Saluki does not run fully on an 8GB card with useful context. You can move some layers to the CPU with a lower -ngl, at a large speed cost. We cover that in can I run Underdog Saluki 27B on 8GB VRAM. On a 12GB card it fits with room to spare.

Ternary Bonsai 2 27B is the 27B that does fit, at 5.54 GiB (PTQ1_0). With a Q8 cache, 16K of context brings the total to 6.22 GiB, and about 28K is the ceiling. It needs the PrismML-Eng llama.cpp fork, which lists CUDA and Metal. Stock llama.cpp, Ollama and LM Studio do not run it, and the fork does not list Vulkan, so skip it on an RX 7600. On Saluki’s own tool-calling test, Bonsai 2 passed 70 of 120 tasks against Saluki’s 88 (Saluki authors’ numbers). Full fit math: Bonsai 2 27B on RTX 3060 12GB.

When 8GB Is the Wrong Card to Buy

If you already own an 8GB card, run Qwen3.5 9B, or Qwen3.6 35B-A3B if you have 32GB of RAM. Do not buy a new 8GB card for local AI. A 12GB card moves you from a 9B model to a 27B model.

What 12GB adds: Qwen3.8 27B at 3 bits (9.40 GiB) with 32K of context, and gpt-oss 20B with only 2 layers offloaded. Details in best local LLM for 12GB VRAM.

Prices as of October 1, 2026, unless marked. They move weekly during the 2026 memory shortage.

Card	VRAM	New	Used
Intel Arc B580	12GB	$329.99 (Newegg, in stock)	$299.99 eBay
RTX 3060	12GB	$419-599+ (Newegg)	$307 average
RTX 5060 Ti	16GB	$589-805 (Aug-Sept 2026 range; $789 at B&H on Sept 24)	n/a

What to buy instead:

Lowest cost new: the Intel Arc B580 12GB. It runs llama.cpp through Vulkan. Its prompt processing is slow: 419 tok/s against 2,013 for an RTX 3060 on the Vulkan scoreboard. See our B580 guide.
CUDA with no setup: a used RTX 3060 12GB at about $307. New, the MSI RTX 3060 12GB starts at $419. Compare its price with the B580 first.
The 16GB step: the RTX 5060 Ti 16GB runs gpt-oss 20B fully on the GPU. NVIDIA also sells an 8GB RTX 5060 Ti. Check the memory size on the listing before you buy. See best local LLM for 16GB VRAM.

We did not verify current prices for the 8GB cards on this page, so we do not quote them.

FAQ

What is the best local LLM for 8GB VRAM?

Qwen3.5 9B at Q4_K_M. Unsloth's file is 5,680,522,464 bytes (5.29 GiB). Its KV cache needs 32 KiB per token at f16, so 32K of context adds 1.0 GiB and the total is about 6.34 GiB. That fits an 8GB card with room for the driver and compute buffers. For agent and coding work, swap in MiMo-V2.6-Distill-Qwen-9B, Xiaomi's fine-tune of the same model, at Q4_K_M (5.44 GiB).

What is the best Qwen model for 8GB VRAM?

Qwen3.5 9B. There is no 9B model in Qwen3.6 or Qwen3.8. Use Q4_K_M (5.29 GiB) for 32K of f16 context or 64K with a Q8 cache. Q5_K_M (6.13 GiB) fits with about 16K of Q8 context. Q6_K (6.95 GiB) and Q8_0 (8.87 GiB) do not fit with useful context. If you have 32GB of system RAM, Qwen3.6 35B-A3B with its experts in RAM is stronger: one tester measured 31.8 tok/s using 6.1 GB of VRAM.

Can 8GB VRAM run gpt-oss 20B?

Not fully on the GPU. The official MXFP4 file is 11.28 GiB, and Unsloth's smallest quant, Q2_K, is still 10.68 GiB. It runs with most of its expert layers in system RAM. The llama.cpp guide shows an RTX 2060 8GB at 32K context with --n-cpu-moe 16. One community user measured 42.32 tok/s on an RTX 4060 Laptop 8GB with --n-cpu-moe 14 and 64GB of DDR5.

Can 8GB VRAM run a 27B model?

Only in a fork, or with almost no context. Underdog Saluki 27B is a 7,532.5 MiB file. llama.cpp keeps its 265 MiB input embedding in CPU RAM, so about 7,257 MiB goes to the card. After the 150 MiB linear-attention state and a 516-839 MiB compute buffer, between 269 MiB and nothing is left for the KV cache and the driver. That is no usable context. Ternary Bonsai 2 27B (5.54 GiB) leaves room for about 28K of Q8 context, but it needs the PrismML llama.cpp fork.

Is 8GB VRAM enough for local LLMs in 2026?

It is enough for a 9B model with 32K context, which handles chat, summaries and simple coding. It is not enough for gpt-oss 20B or a 27B model fully on the GPU. Do not buy a new 8GB card for local AI. As of October 1, 2026, a new Intel Arc B580 12GB was $329.99 at Newegg and a used RTX 3060 12GB averaged $307. A 12GB card runs Qwen3.8 27B at 3 bits with 32K context.

Sources

Hugging Face API file trees (byte sizes), read 2026-10-14: unsloth/Qwen3.5-9B-GGUF, bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF, ggml-org/MiMo-V2.6-Distill-Qwen-9B-GGUF, google/gemma-4-E4B-it-qat-q4_0-gguf, google/gemma-4-12B-it-qat-q4_0-gguf, prism-ml/Ternary-Bonsai-2-27B-gguf, ConwayResearch/Underdog-Saluki-27B-1.0, ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF, ggml-org/gpt-oss-20b-GGUF, unsloth/gpt-oss-20b-GGUF, unsloth/Qwen3.6-35B-A3B-GGUF
Model cards and config.json: Qwen/Qwen3.5-9B, Qwen/Qwen3.8-27B, Qwen/Qwen3.6-35B-A3B, openai/gpt-oss-20b, XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B, ConwayResearch/Underdog-Saluki-27B-1.0, prism-ml/Ternary-Bonsai-2-27B-gguf, read 2026-10-14
llama.cpp discussions #15396 (gpt-oss guide, 8GB examples, driver reserve), #15013 (CUDA scoreboard and comments) and #10879 (Vulkan scoreboard), read 2026-10-14
InsiderLLM, Qwen 3.6 35B MoE guide, firsthand RTX 3060 sweep, read 2026-10-14
NVIDIA RTX 5060 family and RTX 4060 family spec pages (memory size, type and bus width), read 2026-10-14
Prices: OpenClaw DC price reference (Newegg and resaleprices.com reads of 2026-10-01; RTX 5060 Ti range from August-September 2026)