Best Local LLM for 32GB VRAM (2026): Qwen3.8 27B Wins
The best local LLM for 32GB VRAM is Qwen3.8 27B at 6 bits. The 23.56 GiB UD-Q6_K_XL file plus 128K of Q8 context totals 27.95 GiB, which fits any 32GB card. On a slower card, the R9700 or Arc Pro B70, a 35B-A3B mixture-of-experts model decodes 2.7 to 5 times faster, so speed can change the pick.
Running an agent on a 32GB card?
Use the local model calculator first, then see our AI training options. We help you choose a dense or mixture-of-experts model for your card's bandwidth.
Bottom Line
- Best LLM for 32GB VRAM: Qwen3.8 27B at about 6 bits. Use Unsloth’s
UD-Q6_K_XLfile: 23.56 GiB. It runs in stock llama.cpp. - It holds 128K of context. With a Q8 KV cache, weights plus cache total 27.95 GiB. For the full 262K window, use
UD-Q6_K(20.47 GiB): 29.11 GiB total. - Q8_0 (27.05 GiB) fits, but only with about 64K of Q8 context. Most owners get more from 6 bits plus a long window.
- Bandwidth changes the speed, not the fit. An RTX 5090 has 1,792 GB/s. An R9700 has 640 GB/s and an Arc Pro B70 608 GB/s. A dense 27B runs about 2.6x faster on the 5090.
- On an R9700 or B70, consider the MoE. Qwen3.6 35B-A3B decodes 2.7x to 5x faster than a dense 27B on those cards (community tests). It scores lower on Qwen’s own coding benchmarks.
- Nothing above 36B fits. gpt-oss 120B is 59.03 GiB. Laguna S 2.1 is 31.45 GiB even at 1 bit.
Ready to buy? See the tested hardware list with current prices.
How Much of a 32GB Card You Can Use
A “32GB” card has 32 GiB of memory: 32,768 MiB. You cannot fill all of it with the model.
- Driver and runtime context: we assume 0.4 GiB. This is the same assumption as our 12GB page, based on community reports.
- Compute buffers: llama.cpp allocates scratch memory for each batch. In one R9700 test, a 35B-A3B model used 498 MiB of compute buffer at default settings (llama.cpp discussion #21043, community-reported). We assume about 1 GiB to cover larger batches.
- Your desktop: the RTX 5090 is a gaming card, and many owners plug a monitor into it. Check
nvidia-smiorrocm-smibefore you load a model, and subtract what it shows.
That leaves about 30.6 GiB for weights plus KV cache on a headless card. Every “fits” verdict on this page uses that budget.
What Fits in 32GB VRAM
Every file size is a byte count from the Hugging Face API, read 2026-10-13. GiB = 230 bytes. Context totals use the per-token cache sizes worked out below.
| Model | File | Bytes | GiB | Context that fits in ~30.6 GiB | Speed |
|---|---|---|---|---|---|
| Qwen3.8 27B | UD-Q6_K_XL (Unsloth) | 25,299,061,664 | 23.56 | 128K, Q8 cache (27.95 GiB) | About 53 tok/s on a 5090 at short context (our estimate) |
| Qwen3.8 27B | UD-Q6_K (Unsloth) | 21,983,677,344 | 20.47 | Full 262K, Q8 cache (29.11 GiB) | Not measured |
| Qwen3.8 27B | Q8_0 (Unsloth) | 29,047,086,048 | 27.05 | 64K, Q8 cache (29.32 GiB) | About 46 tok/s on a 5090 (our estimate) |
| Qwen3.6 35B-A3B | UD-Q6_K (Unsloth) | 29,308,320,736 | 27.30 | 128K, f16 cache (29.86 GiB) | Not measured |
| Qwen3.6 35B-A3B | UD-Q4_K_M (Unsloth) | 22,134,528,992 | 20.61 | Full 262K, f16 cache (25.67 GiB) | 183 tok/s on a 5090; 54.7 on a B70 (community) |
| Laguna XS 2.1 | Q4_K_M (Poolside) | 20,274,300,032 | 18.88 | Full 262K, f16 cache (28.94 GiB) | Not measured |
| Gemma 4 31B | Q6_K (Unsloth) | 25,201,486,784 | 23.47 | Not calculated | Not measured |
| gpt-oss 20B | MXFP4 (ggml-org) | 12,109,566,624 | 11.28 | Full 128K, f16 cache (14.28 GiB) | Not measured on 32GB cards |
| Laguna S 2.1 | UD-IQ1_S (Unsloth) | 33,766,781,984 | 31.45 | Does not fit | n/a |
| gpt-oss 120B | MXFP4 (ggml-org) | 63,387,346,208 | 59.03 | Does not fit | n/a |
Hardware line: the 32GB cards are the RTX 5090, Radeon AI PRO R9700, Arc Pro B70, RTX PRO 4500 Blackwell and used Instinct MI50 32GB. They all hold the same files. They do not run them at the same speed.
Why Qwen3.8 27B wins
It is the strongest model that fits with room to spare. Qwen’s model card scores it 61.7 on SWE-bench Pro and 73.0 on Terminal Bench 2.1. It scores 90.3 on LiveCodeBench v6. The same card scores Qwen3.6 27B at 53.5, 63.4 and 83.9. The Qwen3.6 35B-A3B card scores the MoE at 49.5 on SWE-bench Pro and 80.4 on LiveCodeBench v6. The two cards use different harnesses, so read the gap as direction, not as an exact margin.
Its cache is small for a 27B. The config.json lists 64 layers. Only every 4th layer uses full attention, with 4 KV heads and a head dimension of 256. The other 48 layers use linear attention with a fixed-size state.
2 (K and V) x 16 full-attention layers x 4 KV heads x 256 head_dim x 2 bytes = 65,536 bytes = 64 KiB per token (f16) Q8_0 cache: 32,768 values per token x 34 bytes per 32 values = 34 KiB per token Fixed linear-attention state: 48 layers x 48 heads x 128 x 128 x 4 bytes = 0.14 GiB
| Context | UD-Q6_K_XL, Q8 cache | UD-Q6_K_XL, f16 cache | Q8_0, Q8 cache | UD-Q6_K, Q8 cache |
|---|---|---|---|---|
| 32K | 24.76 GiB | 25.70 GiB | 28.25 GiB | 21.67 GiB |
| 64K | 25.82 GiB | 27.70 GiB | 29.32 GiB | 22.73 GiB |
| 128K | 27.95 GiB | 31.70 GiB, no | 31.44 GiB, no | 24.86 GiB |
| 192K | 30.07 GiB, just fits | 35.70 GiB, no | 33.56 GiB, no | 26.98 GiB |
| 262K | 32.20 GiB, no | 39.70 GiB, no | 35.69 GiB, no | 29.11 GiB |
This is our arithmetic from the published config. Compare each total with the 30.6 GiB budget above.
One third-party test confirms the method. Hardware Corner ran a 16.68 GiB Q4_K_S file of Qwen3.8 27B on an RTX 5090 at 128K context. It reports 26 GB in use (test page, updated 2026-08-17). Our formula gives 16.68 + 0.14 + 8.0 GiB of f16 cache = 24.82 GiB, plus about 1 GiB of buffers.
Skip the vision projector on long-context runs. The mmproj file adds 0.87 GiB.
How to run it
hf download unsloth/Qwen3.8-27B-GGUF \ Qwen3.8-27B-UD-Q6_K_XL.gguf --local-dir . llama-server -m Qwen3.8-27B-UD-Q6_K_XL.gguf \ -ngl 99 -fa on -c 131072 --parallel 1 \ --cache-type-k q8_0 --cache-type-v q8_0
If the model fails to load, a monitor is probably using part of the card. Drop -c to 98304, or switch to the UD-Q6_K file. For the full window, use UD-Q6_K with -c 262144.
The other picks
Qwen3.6 35B-A3B is a mixture-of-experts model with 35.95B parameters (Hugging Face metadata) and 8 of 256 experts active per token. Its config lists 10 full-attention layers with 2 KV heads of 256 dimensions. That is 20 KiB per token at f16, so the full 262K window costs only 5.0 GiB. Use UD-Q6_K (27.30 GiB) for quality at 128K, or UD-Q4_K_M (20.61 GiB) for the full window and the most speed.
Laguna XS 2.1 (Poolside, 33B total, OpenMDW-1.1 license) has 10 full-attention layers with 8 KV heads of 128 dimensions: 40 KiB per token. The other 30 layers use a 512-token sliding window. The official Q4_K_M file plus the full 262K window totals about 28.9 GiB. Poolside’s own table scores it below Qwen3.6 35B-A3B on all four coding benchmarks it lists, for example 47.6% against 49.5% on SWE-Bench Pro.
gpt-oss 20B is the light pick. Its full 128K context fits in 14.28 GiB, which leaves half the card for a second model. Its cache math is on our 16GB page.
Why the Answer Depends on Bandwidth, Not Only Capacity
All five 32GB cards hold the same files. Decode speed depends on how fast each card reads them. Vendor specifications:
| Card | Memory | Rated bandwidth | Source |
|---|---|---|---|
| RTX 5090 | 32GB GDDR7, 512-bit | 1,792 GB/s | NVIDIA compare page |
| AMD Instinct MI50 32GB | HBM2, 4096-bit | 1 TB/s | AMD spec page |
| RTX PRO 4500 Blackwell | 32GB GDDR7 | 896 GB/s | NVIDIA product page |
| Radeon AI PRO R9700 | 32GB GDDR6, 256-bit | 640 GB/s | AMD product page |
| Arc Pro B70 | 32GB GDDR6, 256-bit | 608 GB/s | Puget Systems spec listing |
A dense model reads every weight for every token. So its decode speed tracks bandwidth. We checked how much of its rated bandwidth each card actually delivers on a dense 27B. The method: measured tok/s x file size in bytes ÷ rated bandwidth.
| Card | Dense 27B test | Measured | Delivered bandwidth | Share of rated |
|---|---|---|---|---|
| RTX 5090 | Qwen3.8 27B, Q4_K_S, 16.68 GiB, 4K context | 74.83 tok/s | 1,340 GB/s | 75% |
| R9700 | Qwen3.5 27B, Q4_K_M, 15.58 GiB, Vulkan | 29.07 tok/s | 486 GB/s | 76% |
| Arc Pro B70 | Qwen3.5 27B, Q4_K_M, 15.6 GiB, SYCL | 20.4 tok/s | 342 GB/s | 56% |
Sources: Hardware Corner (5090), llama.cpp discussion #21043 (R9700), and the PMZFX B70 benchmark repo (B70). All are third-party or community measurements. We did not re-run them.
The 5090 and R9700 deliver the same share of their rated bandwidth. So for a dense 27B, the 5090 is about 2.6x faster, close to its 2.8x bandwidth lead. The B70 delivers less of its bandwidth in that test, which used an April 2026 SYCL build.
Now the non-obvious part: the MoE gap shrinks. A 35B-A3B model reads only its active experts for each token.
| Card | Dense 27B | 35B-A3B MoE | MoE speedup |
|---|---|---|---|
| RTX 5090 | 74.83 tok/s | 183 tok/s (Qwen3.6, UD-Q4_K_M) | 2.4x |
| R9700 | 29.07 tok/s | 147.8 tok/s (Qwen3.5, UD-Q4_K_XL) | 5.1x |
| Arc Pro B70 | 20.4 tok/s | 54.7 tok/s (Qwen3.6, UD-Q4_K_M) | 2.7x |
The 5090 MoE figure is from a community benchmark on zenn.dev (llama.cpp b8870). The R9700 test used the Qwen3.5 model and a slightly smaller file, so treat that row as approximate.
Read across the rows. On a dense 27B, an R9700 runs at 39% of a 5090’s speed. On the MoE, it runs at 81%. As of September 2026, an R9700 cost about 37% of a 5090 ($1,599 street average against the $4,299 floor). The 5090’s premium buys dense-model speed. For MoE models, most of the lead disappears.
What this means for each card, by our estimate for Qwen3.8 27B UD-Q6_K_XL (25.3 GB) at short context:
- RTX 5090: about 53 tok/s (1,340 GB/s ÷ 25.3 GB). The dense 27B is fast enough. Pick it for quality.
- RTX PRO 4500: about 25 tok/s (75% of 896 GB/s ÷ 25.3 GB). We found no dense-model test, so this assumes the same 75% share.
- R9700: about 19 tok/s (486 ÷ 25.3). Usable for chat. For agents that loop, the MoE at 148 tok/s is the better tool.
- Arc Pro B70: about 13 tok/s, scaled from its measured 15.1 tok/s on a 20.9 GiB Qwen3.5 27B Q6_K file. Run the MoE.
- MI50: we found no dense 27B test. On the Llama 2 7B Vulkan scoreboard it delivers only 44% of its rated 1 TB/s, against 91% for the R9700. Its paper bandwidth overstates its speed.
Long context slows every card. In Hardware Corner’s test, the 5090 fell from 74.83 tok/s at 4K to 22.79 tok/s at 128K. In the same series, an RTX 3090 decoded faster than the 5090 at 32K (37.02 against 28.96 tok/s). The test page does not explain this. We report it and do not draw a conclusion.
Which 32GB Card, and Whether to Buy 32GB at All
This page does not repeat the card-buying guide. Read the cheapest 32GB VRAM GPU for software support and stock. The prices below are from our price reference, with the date each was read.
| Card | Price | Read |
|---|---|---|
| RTX 5090 | $4,299 floor; most listings higher | 2026-09-22 |
| RTX PRO 4500 Blackwell | $4,599.99-5,285 | 2026-09-27 |
| Radeon AI PRO R9700 | $1,599 street average, range $1,400-1,900 | 2026-09-24 |
| Arc Pro B70 | $1,299-1,779 | 2026-08-17 |
| Instinct MI50 32GB (used) | About $600-830 eBay asking prices | 2026-09-29 |
- For dense models and the fastest prompt processing: the GIGABYTE RTX 5090 WINDFORCE 32GB. In the tests above, it is the only 32GB card where a dense 27B decodes above 50 tok/s.
- For MoE models at the lowest new price: the Radeon AI PRO R9700. It gets about 81% of a 5090’s MoE speed in the tests above. See best local LLM for R9700.
- The Intel option: the ASRock Intel Arc Pro B70 Creator 32GB. Its software stack is the least settled. See our B70 guide.
- Do not buy an RTX PRO 4500 for local LLMs. It costs more than a 5090 and has half the bandwidth. Buy it only for its 200 W power draw or ECC memory.
- The MI50 is a project card. Read the MI50 guide for its ROCm status before you buy.
Do you need 32GB at all? A 24GB card runs Qwen3.8 27B at 4 bits with 64K of context. A used RTX 3090 24GB averaged $1,486 on 2026-10-01. Buy 32GB for 6-bit quality with 128K context, or for the full 262K window. See best local LLM for 24GB VRAM and is 32GB of VRAM enough?.
What 48GB and 64GB Unlock
48GB runs Qwen3.8 27B at Q8_0 with the full 262K window at f16 cache: 27.05 + 0.14 + 16.0 = 43.19 GiB. It also runs Qwen3.6 35B-A3B at Q8_0 (34.37 GiB), and Laguna S 2.1 at UD-Q2_K_XL (36.96 GiB). The PNY RTX A6000 48GB is the usual single-card route. See the best 48GB VRAM setup.
64GB, usually two 32GB cards, runs Laguna S 2.1 at UD-IQ4_XS (53.61 GiB) split across both. gpt-oss 120B still misses: its 59.03 GiB file exceeds what two cards hold after overhead. Details in best local LLM for 64GB VRAM.
If you already own a 32GB card, do not upgrade for Qwen3.8 27B. It fits at 6 bits with 128K of context. Upgrade for 8-bit weights with a long window, or for 100B-class MoE models.
FAQ
What is the best LLM for 32GB VRAM?
Qwen3.8 27B at about 6 bits per weight. Unsloth's UD-Q6_K_XL file is 25,299,061,664 bytes (23.56 GiB). Its config.json gives a KV cache of 64 KiB per token at f16, because only 16 of its 64 layers use full attention. With a Q8 KV cache, 128K of context adds 4.25 GiB, for a total of 27.95 GiB. That fits a 32GB card with room for the driver and compute buffers. For the full 262K native context, use the UD-Q6_K file (20.47 GiB).
What is the best local LLM for coding on 32GB VRAM?
Qwen3.8 27B. Qwen's model card scores it 61.7 on SWE-bench Pro, 73.0 on Terminal Bench 2.1 and 90.3 on LiveCodeBench v6. Its predecessor Qwen3.6 27B scores 53.5 on SWE-bench Pro on the same card. These are vendor numbers. On slower cards, Qwen3.6 35B-A3B is the best coding LLM for 32GB VRAM if speed matters most. Its own card scores it 49.5 on SWE-bench Pro and 80.4 on LiveCodeBench v6.
What is the best model for 32GB VRAM on an R9700 or Arc Pro B70?
For speed, Qwen3.6 35B-A3B at about 4 bits. These cards have 640 GB/s (R9700) and 608 GB/s (B70) of memory bandwidth, against 1,792 GB/s on an RTX 5090. A dense 27B reads all its weights for every token, so it slows down in proportion. In community llama.cpp tests on an R9700, a 35B-A3B model decoded at 147.8 tok/s and a dense 27B at 29.07 tok/s. On a B70 the figures were 54.7 and 20.4 tok/s. For best quality, Qwen3.8 27B still fits and runs at about 13 to 19 tok/s by our estimate.
What large coding models fit in 32GB VRAM?
The largest useful ones are 31B to 36B. Qwen3.6 35B-A3B fits at UD-Q6_K (27.30 GiB). Laguna XS 2.1 (33B) fits at Q4_K_M (18.88 GiB) with its full 262K context. Gemma 4 31B fits at Q6_K (23.47 GiB). Larger models do not fit. gpt-oss 120B is 59.03 GiB in its official MXFP4 file. Laguna S 2.1 (118B) is 31.45 GiB even at Unsloth's 1-bit UD-IQ1_S, which leaves no room for context. Those need 48GB to 64GB, or expert offload to system RAM.
How much context fits with a 27B model in 32GB VRAM?
With Qwen3.8 27B, up to the full 262K window. Its KV cache is 64 KiB per token at f16 and about 34 KiB with a Q8 cache. Q8_0 weights (27.05 GiB) leave room for about 64K with a Q8 cache. UD-Q6_K_XL (23.56 GiB) holds 128K, and UD-Q6_K (20.47 GiB) holds 262K at 29.11 GiB total. A third-party test measured 26 GB in use at 128K with a 16.68 GiB file, close to our predicted 24.8 GiB plus buffers.
Sources
- Hugging Face API file trees (byte sizes), read 2026-10-13:
unsloth/Qwen3.8-27B-GGUF,unsloth/Qwen3.6-35B-A3B-GGUF,poolside/Laguna-XS-2.1-GGUF,unsloth/gemma-4-31B-it-GGUF,ggml-org/gpt-oss-20b-GGUF,ggml-org/gpt-oss-120b-GGUF,unsloth/Laguna-S-2.1-GGUF,TheBloke/Llama-2-7B-GGUF - Model cards and
config.json, read 2026-10-13:Qwen/Qwen3.8-27B,Qwen/Qwen3.6-35B-A3B,poolside/Laguna-XS-2.1,openai/gpt-oss-20b,google/gemma-4-31B-it - Hardware Corner, Qwen3.8 27B hardware tests (updated 2026-08-17)
- llama.cpp discussion #21043 (R9700 Vulkan tests), #15013 (CUDA scoreboard) and #10879 (Vulkan scoreboard), read 2026-10-13
- PMZFX Arc Pro B70 benchmarks, llm-benchmarks.md (llama.cpp SYCL, build of 2026-04-21); zenn.dev RTX 5090 Qwen3.6 35B-A3B benchmark; both community-reported
- Specifications: NVIDIA GeForce compare page, NVIDIA RTX PRO 4500, AMD Radeon AI PRO R9700, AMD Instinct MI50 32GB spec page, Puget Systems Arc Pro B70 listing, read 2026-10-13
- Prices: OpenClaw DC price reference, rows read 2026-08-17 to 2026-10-01 as dated in the table
See Also
- Best local LLM for RTX 5090: the card-specific picks for the fastest 32GB card
- Best local LLM for R9700: ROCm and Vulkan results for AMD’s 32GB card
- The cheapest 32GB VRAM GPU: which 32GB card to buy, by software support
- Is 32GB of VRAM enough in 2026?: the yes-or-no answer for this tier
- Best local LLM for 32GB RAM: the system-RAM and Mac version of this question
- Best local LLM for 24GB VRAM: the tier below
- Best local LLM for 64GB VRAM: two 32GB cards
- KV cache quantization: Q8 vs Q4: what the Q8 cache costs in quality
- Laguna XS 2.1 on 24GB vs 32GB: the Poolside model in detail
Need OpenClaw fixed live?
Remote rescue sessions for gateway, auth, tunnel, VPS, and model access problems.
See Rescue Session