Hardware October 1, 2026

Best Local LLM for 12GB VRAM (2026): Qwen3.8 27B Wins

On a 12GB card, run Qwen3.8 27B at about 3 bits. The 9.40 GiB file keeps 98.7% of the full model's coding score in its authors' test and leaves room for 32K of context. gpt-oss 20B, the usual 16GB pick, does not fit with any useful context: after its weights, 739 MiB of the card remain, and 32K of its cache needs 768 MiB.

Running an agent on a 12GB card?

Use the local model calculator first, then see our AI training options if you want help sizing the context window so the card does not spill into system RAM.

Bottom Line

Best LLM for 12GB VRAM: Qwen3.8 27B at about 3 bits. Use the ISTA-DASLab GSQ-RCO-IQ3_XXS file: 9.40 GiB. It runs in stock llama.cpp.
It keeps the quality. In its authors’ test, IQ3_XXS scores 84.57 on LiveCodeBench v6 against 85.71 for full BF16. That is 98.7%.
It leaves room for 32K of context with a Q8 KV cache, on a card with no monitor attached. For 48K or more, use the IQ2_S file (8.62 GiB).
gpt-oss 20B does not fit with useful context. After its 11,548.6 MiB of weights, 739 MiB of a 12GB card remain. 32K of its cache needs 768 MiB. Move 2 MoE layers to the CPU and it runs at about 60 tok/s (community-reported).
Simple pick that works in Ollama: Qwen3.5 9B at Q8_0 (8.87 GiB), or Xiaomi’s new MiMo-V2.6 fine-tune of it (same size).
Buying a card? As of October 1, 2026, a new Arc B580 is $329.99 and a used RTX 3060 12GB averages $307. A new 3060 starts at $419. A new RTX 5070 12GB ($869.99) costs more than a 16GB RTX 5060 Ti ($789).

Ready to buy? See the tested hardware list with current prices.

How Much of a 12GB Card You Can Use

A “12GB” card has 12 GiB of memory: 12,288 MiB. You cannot fill all of it with the model.

Driver and CUDA context: one RTX 3060 owner reports 378 MB reserved by the Linux driver on a headless card (llama.cpp discussion #15396, community-reported). We assume 0.4 GiB.
Compute buffers: llama.cpp allocates scratch memory for each batch. We assume about 1 GiB at default batch sizes. This is our assumption, not a measurement. A larger -ub raises it.
Your desktop: if a monitor runs on the same card, Windows or your compositor takes more. Check nvidia-smi before you load a model and subtract what it shows.

That leaves about 10.6 GiB for weights plus KV cache on a headless card. Every “fits” verdict on this page uses that budget.

What Fits in 12GB VRAM

Every file size is a byte count from the Hugging Face API, read 2026-10-01. GiB = 2³⁰ bytes.

Model	File	Size (GiB)	Context that fits in ~10.6 GiB	Speed on a 12GB card
Qwen3.8 27B	GSQ-RCO IQ3_XXS (ISTA-DASLab)	9.40	32K, Q8 cache	About 20-29 tok/s on a 3060 (our estimate)
Qwen3.8 27B	GSQ-RCO IQ2_S (ISTA-DASLab)	8.62	48K, Q8 cache	Not measured
Qwen3.5 9B	Q8_0 (Unsloth)	8.87	32K, f16 cache	Not measured for this file
MiMo-V2.6-Distill-Qwen-9B	Q8_0 (ggml-org)	8.87	32K, f16 cache	Not measured
Gemma 4 12B	QAT Q4_0 (Google)	6.50	Most room of any pick	Not measured
Ternary Bonsai 2 27B	PQ2_0 (Prism ML)	6.71	About 64K (fork only)	Not measured on 12GB
gpt-oss 20B	MXFP4 (ggml-org)	11.28	Under 8K fully on GPU	75 tok/s at 2K; 60 at 32K with `--n-cpu-moe 2` (3060, community)
Qwen3.6 35B-A3B	UD-Q4_K_M (Unsloth)	20.61	Experts in system RAM	38.9 tok/s at 8K, `-ncmoe 24` (3060, InsiderLLM)

Hardware line: the 12GB cards people own or buy are the RTX 3060 12GB, RTX 4070 and 4070 Super, RTX 5070, the laptop RTX 5070 Ti, and the Intel Arc B580. They all hold the same files, because they all hold 12 GiB.

Why Qwen3.8 27B wins

It is the strongest model that fits. Qwen’s own model card scores Qwen3.8 27B at 90.3 on LiveCodeBench v6 and 73.0 on Terminal Bench 2.1. Qwen’s Qwen3.5 9B card scores the 9B at 65.6 and gpt-oss 20B at 74.6 on LiveCodeBench v6. These are separate runs by the same vendor, so read the gap as direction, not as an exact margin.

The 3-bit file keeps that quality. ISTA-DASLab tested its own files against BF16 in one harness:

File	Size	LiveCodeBench v6	GPQA-Diamond
BF16 (their run)	53.8 GB	85.71	89.90
GSQ-RCO IQ3_XXS	9.40 GiB	84.57	88.89
GSQ-RCO IQ2_S	8.62 GiB	82.29	86.36
Unsloth UD-IQ2_S (for comparison)	7.80 GiB	72.00	76.26

These are the authors’ numbers. We did not re-run them. More detail is in our GSQ-RCO quants guide.

Its cache is small for a 27B. The config.json lists 64 layers with full attention on every 4th layer, 4 KV heads and a head dimension of 256. Only 16 layers keep a cache that grows.

2 (K and V) x 16 layers x 4 KV heads x 256 head_dim x 2 bytes = 64 KiB per token (f16)
Q8 cache: about 32 KiB per token
Fixed linear-attention state: 48 layers x 48 heads x 128 x 128 x 4 bytes = 0.14 GiB

Context	Q8 cache	IQ3_XXS total (9.40 + 0.14 + cache)	IQ2_S total (8.62 + 0.14 + cache)
16K	0.5 GiB	10.04 GiB	9.26 GiB
32K	1.0 GiB	10.54 GiB, just fits	9.76 GiB
48K	1.5 GiB	11.04 GiB, no	10.26 GiB
64K	2.0 GiB	11.54 GiB, no	10.76 GiB, just over

This is our arithmetic from the published config. Compare each total with the 10.6 GiB budget above.

Speed is an estimate. On the llama.cpp CUDA scoreboard, an RTX 3060 generates 76.92 tok/s on a 3.56 GiB Llama 7B Q4_0 file. Generation speed scales roughly with the bytes read per token. Scale that to a 9.40 GiB file and you get about 29 tok/s as a ceiling. IQ-type kernels are slower than Q4_0, so we expect about 20 to 29 tok/s on a 3060. Expect about 1.7x that on an RTX 5070, which scores 128.21 on the same scoreboard. These are estimates, not measurements.

One community project shows the upper range with extra work. User kadenball ran Unsloth’s 10.18 GiB UD-IQ3_XXS file on an RTX 3060 at about 29.7 tok/s at 64K context (repo, community-reported). That run uses a custom llama.cpp research patch plus MTP speculative decoding, so stock llama.cpp will not match it.

How to run it

hf download ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF \
  Qwen3.8-27B-GSQ-RCO-IQ3_XXS.gguf --local-dir .

llama-server -m Qwen3.8-27B-GSQ-RCO-IQ3_XXS.gguf \
  -ngl 99 -fa on -c 32768 --parallel 1 \
  --cache-type-k q8_0 --cache-type-v q8_0

Skip the vision projector on 12GB. The mmproj file adds 0.87 GiB. If the model fails to load, drop -c to 16384 or switch to the IQ2_S file.

The 9B picks

Qwen3.5 9B is the newest small Qwen. There is no 9B in Qwen3.6 or Qwen3.8. Its config lists 32 layers with full attention on every 4th layer and 4 KV heads of 256 dimensions, so the f16 cache is 32 KiB per token. At Q8_0 (8.87 GiB), 32K of context adds 1.0 GiB. Use Q6_K (6.95 GiB) if you want 64K or more.

MiMo-V2.6-Distill-Qwen-9B (published 2026-09-21) is Xiaomi’s fine-tune of Qwen3.5 9B on agent and coding data. Xiaomi’s card scores it 37.1 on Terminal Bench 2.1 against 27.0 for the base 9B, and 61.1 against 60.0 on SWE Verified. These are vendor numbers. It has the same architecture and the same size, so it is a free swap if you run agents.

Gemma 4 12B QAT is Google’s quantization-aware trained Q4_0 file at 6.50 GiB. It leaves the most room of any good pick, for long documents or a second model.

The fork option: Ternary Bonsai 2 27B

Prism ML’s 1.72-bit build of Qwen3.8 27B is 6.71 GiB (PQ2_0) or 5.54 GiB (PTQ1_0). It needs the PrismML-Eng llama.cpp fork; the latest release was prism-b10743-adfffbe on 2026-09-25. Upstream support is still an open request (issue #29058, checked 2026-10-01). Ollama and LM Studio do not run it. The fork lists CUDA, not Vulkan, so skip it on an Arc B580. Full fit math is in our Bonsai 2 27B on RTX 3060 guide.

Why gpt-oss 20B Is Not the 12GB Answer

gpt-oss 20B is our 16GB pick. OpenAI’s model card says it runs “within 16GB of memory”. On 12GB the arithmetic fails.

Card:                       12,288 MiB
MXFP4 weights:             -11,548.6 MiB   (12,109,566,624 bytes)
Left before anything else:     739.4 MiB

KV cache, from config.json: 12 full-attention layers x 2 x 8 KV heads x 64 head_dim x 2 bytes
                          = 24 KiB per token
32K context:                   768 MiB   (more than what is left)

So even with zero driver memory and zero compute buffers, 32K of context does not fit. After the 0.4 GiB driver reserve, you have room for a few thousand tokens. One RTX 3060 owner measured 75 tok/s at a 2,048-token context with everything on the GPU (community-reported, discussion #15396).

Smaller quants do not rescue it. OpenAI trained the expert weights in 4-bit MXFP4. Re-quantizing them saves little. Unsloth’s files, read from the Hugging Face API:

gpt-oss 20B file	Size
MXFP4 (official, ggml-org)	11.28 GiB
Unsloth Q4_K_M	10.83 GiB
Unsloth Q2_K	10.68 GiB

Dropping to 2 bits saves 0.6 GiB. Moving 2 MoE layers to the CPU is the better route.

The MoE-Offload Route for Bigger Models

Mixture-of-experts models read only a few experts per token. llama.cpp’s --n-cpu-moe N flag keeps the expert weights of N layers in system RAM. Attention and shared weights stay on the GPU. Full flag guide: llama.cpp MoE offload flags explained.

gpt-oss 20B with 2 layers offloaded. Both results are community-reported in discussion #15396:

Card	Context	Flags	Speed
RTX 5070 12GB, DDR5-6000	32K	`--n-cpu-moe 2`, `-ub 2048`	62.21 tok/s
RTX 3060 12GB, DDR4	32K	`-ncmoe 2`, `-ub 512` (2048 did not fit)	60 tok/s
RTX 3060 12GB, DDR4	16K	`-ncmoe 2`	64 tok/s
RTX 3060 12GB	2K	none, all on GPU	75 tok/s
RTX 5070 12GB	2K	none, all on GPU	128 t/s

Look at the 32K rows. With nothing offloaded, the RTX 5070 is about 1.7x faster than the 3060. With 2 layers in system RAM, the two cards run at the same speed. The bottleneck moves to the CPU and system RAM. If you plan to offload, a faster 12GB card buys you almost nothing.

llama-server -hf ggml-org/gpt-oss-20b-GGUF \
  --ctx-size 32768 --jinja -ngl 99 -fa on \
  --n-cpu-moe 2 -ub 512

Qwen3.6 35B-A3B at 4 bits. The UD-Q4_K_M file is 20.61 GiB. InsiderLLM measured 38.9 tok/s at 8K context on an RTX 3060 12GB with -ncmoe 24 and 32GB of DDR4-2133. The same rig decoded 35.8 tok/s with a 16,128-token prompt (source, firsthand, updated 2026-09-30). You need at least 32GB of system RAM. Qwen’s card scores this model 80.4 on LiveCodeBench v6, below the dense Qwen3.8 27B, but it is faster and its cache is only 20 KiB per token.

Which 12GB Card to Buy, and Whether to Buy 12GB at All

Prices as of October 1, 2026, unless marked. They move weekly during the 2026 memory shortage.

Card	New	Used	Source
Intel Arc B580 12GB	$329.99 (ASRock Challenger, Newegg, in stock); $369.99 Amazon (Sept 30)	$299.99 eBay	newegg.com; videocardprices.com
RTX 3060 12GB	$419-599+	$307 average (198 listings)	newegg.com; resaleprices.com
RTX 5070 12GB	$869.99-1,149.99 (Sept 29)	n/a	newegg.com
RTX 4070 / 4070 Super 12GB	Not verified	Not verified	n/a
RTX 5060 Ti 16GB	$789 at B&H (Sept 24)	n/a	gpuprix tracker
RTX 3090 24GB	n/a	$1,463 average (Sept 24)	resaleprices.com

Two prices are inverted in 2026.

A new RTX 3060 12GB costs more than a new Arc B580 and more than a used 3060. The cheapest new 3060 is $419; a used one averages $307. NVIDIA relaunched the card during the shortage, and it now sells well above its 2021 launch price.
A new RTX 5070 12GB costs more than a new RTX 5060 Ti 16GB. $869.99 against $789, both September 2026 reads. The 5060 Ti holds gpt-oss 20B fully on the GPU. The 5070 cannot.

What to buy:

Lowest cost, CUDA, no setup: a used RTX 3060 12GB at about $307. If you want new, the MSI RTX 3060 12GB is the card in most community tests on this page. Compare its price with the B580 first.
Lowest cost new: the Intel Arc B580 12GB. It has 456 GB/s of bandwidth (Intel) against the 3060’s 360 GB/s (192-bit at 15 Gbps). On the Vulkan scoreboard, with flash attention on, it generates 72.00 tok/s against 80.59 for the 3060. Prompt processing is the catch: 419 against 2,013 tok/s, about 4.8x slower. For agents that read long files, that delay adds up. See our B580 guide.
Do not buy an RTX 5070 for LLMs at these prices. Its 12GB holds the same files as a 3060. For less money, a 16GB card holds more.
Laptops: NVIDIA lists the laptop RTX 5070 Ti with 12GB, and the laptop RTX 5070 with 12GB or 8GB. Check the exact model before you buy.

What 16GB and 24GB Unlock

16GB runs gpt-oss 20B fully on the GPU with 32K to 64K of context, and Qwen3.8 27B at 3.5 bits (IQ3_S, 10.96 GiB) with 64K. On 12GB you get the 3-bit or 2.75-bit file instead. The RTX 5060 Ti 16GB is the cheapest new 16GB NVIDIA card. Details in best local LLM for 16GB VRAM.

24GB runs Qwen3.8 27B at 4 bits (UD-Q4_K_XL) with 64K of f16 context. That is the first tier with no compromise on the 27B. A used RTX 3090 24GB averaged $1,463 in September 2026, about 4.8x a used 3060. See Qwen 3.8 27B on RTX 3090.

If you already own a 12GB card, do not upgrade for the 27B. The 3-bit file gets 98.7% of the coding score in its authors’ test. Upgrade if you need more than 32K of context on a dense 27B, or gpt-oss 20B at full speed.

FAQ

What is the best LLM for 12GB VRAM?

Qwen3.8 27B at about 3 bits per weight. The ISTA-DASLab GSQ-RCO IQ3_XXS file is 10,094,357,632 bytes (9.40 GiB). In its authors' own test it scores 84.57 on LiveCodeBench v6 against 85.71 for the full BF16 model. With a Q8 KV cache, 32K of context adds about 1.0 GiB, which fits a 12GB card if no monitor runs on it. For longer context, use the IQ2_S file (8.62 GiB). For a small, simple model that runs in Ollama, use Qwen3.5 9B or its MiMo-V2.6 fine-tune at Q8_0 (8.87 GiB).

Can 12GB VRAM run a 27B model?

Yes, at 2 to 3 bits per weight. Qwen3.8 27B is 9.40 GiB at IQ3_XXS and 8.62 GiB at IQ2_S in the ISTA-DASLab GSQ-RCO repo, and both run in stock llama.cpp. Ternary Bonsai 2 27B is smaller still at 6.71 GiB, but it needs the PrismML llama.cpp fork. The usual 4-bit 27B files are 14.3 GiB or larger and do not fit.

Can 12GB VRAM run gpt-oss 20B?

Only with part of it in system RAM. The official MXFP4 file is 12,109,566,624 bytes, or 11,548.6 MiB. A 12GB card has 12,288 MiB, so 739 MiB remain before the driver, buffers or any context. Its KV cache needs 24 KiB per token, so 32K of context alone needs 768 MiB. With --n-cpu-moe 2 at 32K context, community users in llama.cpp discussion #15396 measured 60 tok/s on an RTX 3060 and 62 tok/s on an RTX 5070. Smaller quants barely help: Unsloth's Q2_K is still 10.68 GiB.

Is 12GB VRAM enough for local LLMs in 2026?

Yes for one good model at a time: a 27B at 3 bits with 32K context, a 9B at 8 bits, or a 35B mixture-of-experts model with experts in system RAM. It is not enough for 4-bit 27B models or for gpt-oss 20B fully on the GPU. If you are buying new, check 16GB first: in September 2026 an RTX 5060 Ti 16GB was $789 at B&H, while the cheapest RTX 5070 12GB was $869.99 at Newegg.

Which 12GB GPU is best for local LLMs?

For the lowest cost, an Intel Arc B580 at $329.99 new at Newegg or a used RTX 3060 12GB at about $307 average, both as of October 1, 2026. The RTX 3060 runs CUDA with no setup. The B580 runs llama.cpp through Vulkan and generates at a similar speed, but its prompt processing on the Vulkan scoreboard is about 4.8x slower than the 3060's. A new RTX 3060 at $419 or more is poor value. The RTX 5070 is about 1.7x faster than a 3060 but costs more than a 16GB RTX 5060 Ti.

Sources

Hugging Face API file trees (byte sizes), read 2026-10-01: ggml-org/gpt-oss-20b-GGUF, unsloth/gpt-oss-20b-GGUF, ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF, unsloth/Qwen3.8-27B-GGUF, unsloth/Qwen3.5-9B-GGUF, ggml-org/MiMo-V2.6-Distill-Qwen-9B-GGUF, google/gemma-4-12B-it-qat-q4_0-gguf, prism-ml/Ternary-Bonsai-2-27B-gguf, unsloth/Qwen3.6-35B-A3B-GGUF
Model cards and config.json: openai/gpt-oss-20b, Qwen/Qwen3.8-27B, Qwen/Qwen3.5-9B, Qwen/Qwen3.6-35B-A3B, XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B, ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF, read 2026-10-01
llama.cpp discussions #15396 (gpt-oss guide and 12GB community runs), #15013 (CUDA scoreboard) and #10879 (Vulkan scoreboard), read 2026-10-01
InsiderLLM, Qwen 3.6 35B MoE guide (updated 2026-09-30); kadenball/qwen38-27b-rtx3060-dcfr on GitHub; both community-reported
Intel Arc B580 specifications (intel.com); NVIDIA RTX 5070 family and RTX 50 laptop spec pages, read 2026-10-01
Prices: newegg.com search and resaleprices.com, 2026-10-01; videocardprices.com, 2026-09-30; OpenClaw DC price reference for the RTX 5070, RTX 5060 Ti and RTX 3090 rows (September 2026)