Hardware October 13, 2026

Best Local LLM for 24GB VRAM (2026): Qwen3.8 27B Wins

On any 24GB card, run Qwen3.8 27B at 4 bits. The 16.35 GiB file leaves room for about 96K of context at full precision, or 128K with a Q8 cache. It is the strongest model that fits with real context. gpt-oss 20B is about four times faster and holds its full 128K window, so it is the speed pick. Qwen3.6 35B-A3B fits too, and it holds the same context as the 27B.

Running an agent on a 24GB card?

Use the local model calculator first, then see our AI training options if you want help picking the quant and context so the model stays on the card.

Bottom Line

Best LLM for 24GB VRAM: Qwen3.8 27B at 4 bits. Use Unsloth’s UD-Q4_K_XL file: 16.35 GiB. It runs in stock llama.cpp.
It holds real context. About 96K tokens with an f16 cache, or 128K with a Q8 cache, on a card with no monitor attached.
It is the strongest model that fits. Qwen scores it 90.3 on LiveCodeBench v6 and 73.0 on Terminal Bench 2.1 (vendor numbers).
Speed pick: gpt-oss 20B. Weights plus its full 128K window total 14.3 GiB. It generates 161.77 tok/s on a 3090 (community-reported). Qwen3.8 27B gets about 40.
MoE pick: Qwen3.6 35B-A3B. Its file is 4.26 GiB bigger than the 27B’s, yet it holds the same context. Its cache costs 20 KiB per token, not 64.
The card does not change the model. A 3090, 4090, RX 7900 XTX and Tesla P40 all hold the same files. They differ in speed, price and software.
Buying a card? On October 13, 2026, a used RTX 3090 averaged $1,518. A used RTX 4090 had a $3,000 median. That is about 2x the money for 14% faster replies.

Ready to buy? See the tested hardware list with current prices.

How Much of a 24GB Card You Can Use

A “24GB” card has 24 GiB of memory: 24,576 MiB. You cannot fill all of it with the model.

Driver and CUDA context: we assume 0.4 GiB, the same reserve we use on our 12GB page. This is an assumption.
Compute buffers: llama.cpp allocates scratch memory for each batch. We assume about 1 GiB at default batch sizes. A larger -ub raises it.
Your desktop: if a monitor runs on the same card, Windows or your compositor takes more. Check nvidia-smi (or rocm-smi) before you load a model.

That leaves about 22.6 GiB for weights plus KV cache on a headless card. Every “fits” verdict on this page uses that budget.

A third-party measurement supports the budget. Hardware Corner loaded a 16.68 GiB Qwen3.8 27B Q4_K_S file. It reported about 22 GB at 64K and about 26 GB at 128K (source, updated 2026-08-17). Our method gives 20.8 GiB and 24.8 GiB for weights plus cache. Add about 1 GiB of buffers and the two agree.

What Fits in 24GB VRAM

Every file size is a byte count from the Hugging Face API, read 2026-10-13. GiB = 2³⁰ bytes. Cache sizes are our arithmetic from each model’s config.json.

Model	File	Size (GiB)	KV cache per token (f16)	Context that fits in ~22.6 GiB	Measured speed on 24GB
Qwen3.8 27B (dense)	UD-Q4_K_XL (Unsloth)	16.35	64 KiB	About 96K f16; 128K with Q8 cache	40.31 tok/s on a 3090, 46.16 on a 4090 at 4K (16.68 GiB Q4_K_S, Hardware Corner)
Qwen3.8 27B	UD-IQ4_XS (Unsloth)	13.27	64 KiB	128K f16; 262K with Q8 cache, just	Not measured
Qwen3.8 27B	UD-Q5_K_M (Unsloth)	18.41	64 KiB	32K f16; 64K with Q8 cache	Not measured
Qwen3.6 27B (dense)	Q4_K_M (Unsloth)	15.66	64 KiB	About 110K f16	Not measured for this file
gpt-oss 20B (MoE)	MXFP4 (ggml-org)	11.28	24 KiB	Full 128K with 8.3 GiB spare	161.77 tok/s on a 3090, 221.95 on a 4090 (#15396)
Qwen3.6 35B-A3B (MoE)	UD-Q4_K_M (Unsloth)	20.61	20 KiB	About 96K f16	Not measured for this file
Qwen3.6 35B-A3B	UD-IQ4_NL_XL (Unsloth)	18.16	20 KiB	89,600 measured (CUDA)	139.6 tok/s on a 3090 (Giles Thomas)
Qwen3.6 35B-A3B	UD-IQ4_XS (Unsloth)	16.51	20 KiB	Full 262K f16	Not measured
Laguna XS 2.1 (MoE)	Q4_K_M (ggml-org)	18.22	40 KiB	About 110K f16	Not measured
Gemma 4 31B (dense)	QAT UD-Q4_K_XL (Unsloth)	16.10	Not computed	Not computed	Not measured

Notes on the table:

Gemma 4 31B stores keys and values in an unusual layout (attention_k_eq_v in its config). We did not confirm how llama.cpp sizes that cache, so we give no context figure.
What does not fit: Qwen3.8 27B at Q8_0 is 27.05 GiB. Qwen3.6 35B-A3B at Q8_0 is 34.37 GiB. Qwen3.8-Flash-Next has about 180 billion parameters, so it is not a 24GB model. For 70B models, see can 24GB VRAM run a 70B LLM.

Why Qwen3.8 27B wins

It is the strongest model that fits with real context. Qwen’s Qwen3.8 27B model card compares it with the older Qwen3.6 27B and Meta’s Muse Glimmer 30B:

Benchmark (Qwen3.8 27B model card)	Qwen3.8 27B	Qwen3.6 27B	Muse Glimmer 30B
Terminal Bench 2.1	73.0	63.4	51.7
SWE-bench Pro	61.7	53.5	51.2
GPQA Diamond	89.2	87.8	83.5
LiveCodeBench v6	90.3	83.9	not listed

Qwen’s Qwen3.6 35B-A3B card scores that MoE at 80.4 on LiveCodeBench v6 and 49.5 on SWE-bench Pro. Qwen’s Qwen3.5 9B card scores gpt-oss 20B at 74.6 on LiveCodeBench v6. These are vendor numbers from separate cards, so read the gaps as direction, not exact margins.

It replaces Qwen3.6 27B at no memory cost. Both models list the same config.json values and 27,781,427,952 parameters on Hugging Face. The files are within a few hundred MiB of each other. If you run Qwen3.6 27B today, swap the file and keep your flags.

Its cache is small for a 27B. The config.json lists 64 layers. 48 are linear-attention layers with a fixed-size state. Only 16 are full attention, each with 4 KV heads of 256 dimensions.

f16 cache:  2 (K and V) x 16 layers x 4 KV heads x 256 head_dim x 2 bytes = 64 KiB per token
Q8 cache:   q8_0 stores 34 bytes per 32 values, so 64 KiB x 34/64      = 34 KiB per token
Fixed state: 48 layers x 48 value heads x 128 x 128 x 4 bytes           = 0.14 GiB

Context	Cache	UD-Q4_K_XL total (16.35 + 0.14 + cache)	UD-IQ4_XS total (13.27 + 0.14 + cache)
64K	f16, 4.0 GiB	20.49 GiB, fits	17.41 GiB
96K	f16, 6.0 GiB	22.49 GiB, just fits	19.41 GiB
128K	f16, 8.0 GiB	24.49 GiB, no	21.41 GiB, fits
128K	Q8, 4.25 GiB	20.74 GiB, fits	17.66 GiB
262K (native max)	Q8, 8.5 GiB	24.99 GiB, no	21.91 GiB, just fits

Compare each total with the 22.6 GiB budget. On paper, a 24GB card can hold this model’s full 262K window. That needs the IQ4_XS file, a Q8 cache and a card with no monitor. Test it before you rely on it.

Speed is measured, by a third party. Hardware Corner used llama.cpp build 153d324bc with a 16.68 GiB Q4_K_S file:

Card	Context	Prompt	Reply
RTX 3090	4K	1,308.05 tok/s	40.31 tok/s
RTX 3090	64K	766.59 tok/s	33.95 tok/s
RTX 4090	4K	2,962.59 tok/s	46.16 tok/s
RTX 4090	64K	1,918.13 tok/s	38.41 tok/s

The model also has one multi-token-prediction layer (mtp_num_hidden_layers: 1). Unsloth ships it as a separate 1.28 GiB file. Budget that 1.28 GiB if you turn MTP on. The measured gains are in our Qwen3.8 27B on RTX 3090 guide.

How to run it

hf download unsloth/Qwen3.8-27B-GGUF \
  Qwen3.8-27B-UD-Q4_K_XL.gguf --local-dir .

llama-server -m Qwen3.8-27B-UD-Q4_K_XL.gguf \
  -ngl 99 -fa on -c 65536 --parallel 1

For 128K, add --cache-type-k q8_0 --cache-type-v q8_0 and set -c 131072. Vision needs the mmproj-F16 file too: 0.86 GiB more. For more than 128K, switch to UD-IQ4_XS. For how the 4-bit files differ, see IQ4_XS vs Q4_K_M vs Q4_K_XL.

The MoE Holds the Same Context as the Dense 27B

Most guides say a 35B MoE on 24GB leaves “too little room for context”. Free memory is the wrong measure. Context depends on free memory divided by cache cost per token.

Qwen3.6 35B-A3B has 40 layers. Only 10 are full attention, each with 2 KV heads of 256 dimensions. Here is the same arithmetic for both 4-bit picks:

MoE cache:   2 x 10 layers x 2 KV heads x 256 x 2 bytes = 20 KiB per token
MoE state:   30 layers x 32 heads x 128 x 128 x 4 bytes = 0.06 GiB

Dense 27B, UD-Q4_K_XL:   22.6 - 16.35 - 0.14 = 6.11 GiB free / 64 KiB = ~100,100 tokens
MoE 35B-A3B, UD-Q4_K_M:  22.6 - 20.61 - 0.06 = 1.93 GiB free / 20 KiB = ~101,200 tokens

The MoE has about one third of the free memory, but its cache costs about one third as much per token. Both land near 100K tokens. We round both down to “about 96K” for safety.

A real run confirms the scale. Giles Thomas loaded the 18.16 GiB UD-IQ4_NL_XL file on one RTX 3090 with nothing offloaded. With CUDA, llama.cpp fit 89,600 tokens and generated 139.6 tok/s, with prompts at 3,360.4 tok/s (source, 2026-07-24, community-reported).

The backend changes the context, too. On the same card and file, his Vulkan build fit only 51,200 tokens at 122.4 tok/s. That matters on an RX 7900 XTX, where Vulkan is a common route. Plan for less context than the CUDA figures on this page.

Pick the MoE if you want about 3.5x the 27B’s reply speed and accept lower scores. Qwen’s own cards give 80.4 against 90.3 on LiveCodeBench v6. The full trade-off is in MoE vs dense on a 24GB card.

One note on Laguna XS 2.1. Poolside’s 33B MoE fits at 18.22 GiB, with a 40 KiB per-token cache. But Poolside’s own model card scores Qwen3.6 35B-A3B higher on all four benchmarks it lists. On SWE-bench Verified it is 73.4% against 70.9%. Its license is OpenMDW-1.1, not Apache 2.0. On 24GB we would run the Qwen. More in Laguna XS 2.1 on 24GB vs 32GB.

When to Run gpt-oss 20B Instead

gpt-oss 20B is our 16GB pick. On 24GB it has no constraint.

Budget:                     ~22.6 GiB
MXFP4 weights:              -11.28 GiB   (12,109,566,624 bytes)
KV cache, from config.json: 12 full-attention layers x 2 x 8 KV heads x 64 head_dim x 2 bytes
                          = 24 KiB per token
Full 131,072-token window:   -3.0 GiB
Left over:                   ~8.3 GiB

The other 12 layers use a 128-token sliding window, so their cache stays tiny. The spare 8.3 GiB holds a second full-length slot (--parallel 2) for two agents at once.

It is about four times faster than the 27B. In discussion #15396, with flash attention on (community-reported):

Card	Reply (tg128)	Prompt at 32K
RTX 3090	161.77 tok/s	3,577.10 tok/s
RTX 4090	221.95 tok/s	5,112.35 tok/s

Pick gpt-oss 20B if you run a chat UI where reply speed matters most, or you want two agent slots on one card. Pick Qwen3.8 27B if the agent writes code or runs terminal tasks. For the bigger sibling, see gpt-oss 120B vs 20B.

llama-server -hf ggml-org/gpt-oss-20b-GGUF \
  --ctx-size 0 --jinja -ub 2048 -b 2048

That is the command from the llama.cpp guide for cards under 64GB. --ctx-size 0 uses the model’s full window.

Which 24GB Card to Buy

All four cards hold the same files. They differ in speed, software and price. Speeds below are from the llama.cpp scoreboards, on a Llama 2 7B Q4_0 file with flash attention on (community-reported).

Card	Reply, CUDA	Prompt, CUDA	Reply, Vulkan	Prompt, Vulkan	Price (source, date)
RTX 3090 (used)	161.89 tok/s	5,560.06 tok/s	171.61 tok/s	5,200.68 tok/s	$1,518 avg, 315 listings (resaleprices.com, Oct 13, 2026)
RTX 4090 (used)	188.96 tok/s	14,770.63 tok/s	190.10 tok/s	10,830.41 tok/s	$3,000 median, 26 listings (resaleprices.com, Oct 13, 2026)
RX 7900 XTX (new)	n/a (no CUDA)	n/a	190.92 tok/s	3,889.54 tok/s	$1,129.99 Amazon (videocardprices.com, data dated Sept 26, 2026)
Tesla P40 (used)	53.73 tok/s	1,079.66 tok/s	59.22 tok/s	484.37 tok/s	$240-350 (as of August 2026)

CUDA figures are from discussion #15013. Vulkan figures are from discussion #10879. Prices move weekly during the 2026 memory shortage.

A used RTX 3090 now costs more than it did new. Its launch price in September 2020 was $1,499. The October 13 used average is $1,518.

What to buy:

Most people: a used RTX 3090. It runs CUDA with no setup and writes Qwen3.8 27B at about 40 tok/s. Read how to buy a used RTX 3090 safely first. The EVGA RTX 3090 24GB is a common model.
Coding agents with long prompts: an RTX 4090. On Qwen3.8 27B it reads prompts 2.26x faster and writes replies 14% faster than a 3090. At these used prices it costs about 2x as much. The GIGABYTE RTX 4090 24GB is the model we link. Full comparison: RTX 3090 vs 4090.
New card, no CUDA needed: the RX 7900 XTX. On the Vulkan scoreboard it writes about 11% faster than a 3090 and reads prompts about 25% slower. We found no clean Qwen3.8 27B measurement on it. At about $1,130 new, it costs less than a used 3090. Setup notes: best local LLM for RX 7900 XTX.
Lowest cost, with eyes open: the Tesla P40. It is the cheapest 24GB card. It writes at about a third of a 3090’s speed on the scoreboard. Scaled from the 3090’s 40.31 tok/s, expect about 13 tok/s on Qwen3.8 27B. That is our estimate, not a measurement. CUDA 13 removed Pascal support, so you stay on CUDA 12.x builds. It is passively cooled, so you need a fan kit. The Tesla P40 24GB suits a homelab, not a daily coding rig. Read is the Tesla P40 worth it first.

What 32GB and 48GB Unlock

32GB runs Qwen3.8 27B at 5 or 6 bits with long context. UD-Q6_K is 20.47 GiB, too big for 24GB with useful context. It also runs Qwen3.6 35B-A3B at 4 bits with its full 262K window. The RTX 5090 32GB wrote 74.83 tok/s on Qwen3.8 27B at 4K in the same Hardware Corner test. That is 1.86x a 3090. Its floor price was $4,299 on September 22, 2026, about 2.8x a used 3090. Details in best local LLM for 32GB VRAM.

48GB, usually two used 3090s, runs Qwen3.8 27B at Q8_0 (27.05 GiB) with long context. It also leaves room for two models at once. See dual RTX 3090 vs RTX 5090.

If you own a 24GB card, do not upgrade for the 27B. The 4-bit file with 64K to 128K of context covers most agent work. Upgrade for higher-bit files or two models at once.

One tier down: on 16GB, the 4-bit 27B files do not fit. See best local LLM for 16GB VRAM.

FAQ

What is the best LLM for 24GB VRAM?

Qwen3.8 27B at 4 bits. Unsloth's UD-Q4_K_XL file is 17,559,178,144 bytes (16.35 GiB). Only 16 of its 64 layers keep a growing KV cache, so the cache costs 64 KiB per token in f16. On a headless 24GB card that leaves room for about 96K tokens, or 128K with a Q8 cache. Hardware Corner measured 40.31 tok/s on an RTX 3090 and 46.16 tok/s on an RTX 4090 at 4K context with a 16.68 GiB Q4_K_S file. If reply speed matters more than quality, run gpt-oss 20B.

What is the best Qwen model for 24GB VRAM?

Qwen3.8 27B for quality: UD-Q4_K_XL (16.35 GiB) for up to about 96K of f16 context, or UD-IQ4_XS (13.27 GiB) for 128K and more. Qwen3.6 35B-A3B for speed: UD-Q4_K_M (20.61 GiB) also holds about 96K, because its cache is only 20 KiB per token. Qwen's own model cards score Qwen3.8 27B at 90.3 on LiveCodeBench v6 and Qwen3.6 35B-A3B at 80.4.

Can Qwen 3.6 27B run on 24GB VRAM?

Yes. Unsloth's Qwen3.6 27B Q4_K_M file is 15.66 GiB, and 64K of f16 context adds 4.0 GiB. But Qwen3.8 27B has the same config.json and the same 27,781,427,952 parameters, so it uses the same memory. On Qwen's Qwen3.8 27B model card it scores 73.0 against 63.4 on Terminal Bench 2.1 and 90.3 against 83.9 on LiveCodeBench v6. Swap to Qwen3.8 27B.

Which 24GB VRAM LLM is the fastest?

gpt-oss 20B. The official MXFP4 file is 12,109,566,624 bytes (11.28 GiB), and its full 131,072-token window adds 3.0 GiB of cache. In llama.cpp discussion #15396 it generated 161.77 tok/s on an RTX 3090 and 221.95 tok/s on an RTX 4090. Qwen3.8 27B generates about 40 tok/s on a 3090, but it scores much higher on coding tests.

What is the cheapest 24GB GPU?

The used Tesla P40, at about $240 to $350 as of August 2026. It is about a third as fast as an RTX 3090, it has no fan, and CUDA 13 cannot build for it. The cheapest practical 24GB card is a used RTX 3090: $1,518 average across 315 eBay listings on October 13, 2026 (resaleprices.com). A new RX 7900 XTX was $1,129.99 at Amazon in tracker data dated September 26, 2026. It needs ROCm or Vulkan instead of CUDA.

Sources

Hugging Face API file trees (byte sizes), read 2026-10-13: unsloth/Qwen3.8-27B-GGUF, unsloth/Qwen3.6-27B-GGUF, ggml-org/gpt-oss-20b-GGUF, unsloth/Qwen3.6-35B-A3B-GGUF, ggml-org/Laguna-XS-2.1-GGUF, unsloth/gemma-4-31B-it-qat-GGUF
Model cards and config.json, read 2026-10-13: Qwen/Qwen3.8-27B, Qwen/Qwen3.6-27B, Qwen/Qwen3.6-35B-A3B, Qwen/Qwen3.5-9B (gpt-oss 20B score), openai/gpt-oss-20b, poolside/Laguna-XS-2.1, Qwen/Qwen3.8-Flash-Next (parameter count only)
llama.cpp discussions #15013 (CUDA scoreboard), #10879 (Vulkan scoreboard) and #15396 (gpt-oss guide), read 2026-10-13
Hardware Corner, We Tested Qwen3.8 27B, updated 2026-08-17
Giles Thomas, Benchmarking Qwen 3.6 35B MoE on an RTX 3090, 2026-07-24, community-reported
NVIDIA Developer Blog, Navigating GPU architecture support (CUDA 13 drops Maxwell, Pascal and Volta)
Prices: resaleprices.com RTX 3090 and RTX 4090 pages, updated 2026-10-13; videocardprices.com RX 7900 XTX page (data dated 2026-09-26), read 2026-10-13; OpenClaw DC price reference for the Tesla P40 (August 2026), RTX 5090 (September 2026) and RTX 3090 launch price rows