What is the best local LLM for an RTX 5090?

Qwen 3.6 35B-A3B (MoE) at Q6_K is the best pick that takes advantage of the 5090's 32 GB VRAM. It uses about 28 GB with comfortable context and runs at ~80 tokens/sec thanks to the MoE design + 1792 GB/s bandwidth. For agentic coding, Laguna XS 2.1 at Q4_K_M (20 GB, 33B MoE with 3B active params, 70.9% SWE-Bench Verified) is the July 2026 pick. For OpenClaw, gpt-oss 20B at Q8_0 (~22 GB) is the safer production pick.

Can the RTX 5090 run Laguna XS 2.1?

Yes, comfortably. laguna-xs-2.1:q4_K_M is 20 GB, so the 5090's 32 GB VRAM leaves about 12 GB of headroom — enough KV cache for roughly 64K context, far more than the 8-16K cap a 24 GB card forces. The Q8 weights (~33 GB) do not fit 32 GB, so stay on Q4. Because only 3B parameters are active per token, generation runs at small-model speed despite the 33B total size.

Can the RTX 5090 run Gemma 4 31B?

Yes, and the 5090 is the first consumer card that runs it well. Gemma 4 31B is Google's dense 30.7B model with a 256K context window; Q4_K_M is about 18.3 GB and Q6_K is about 25.2 GB, so the 5090's 32 GB fits Q6_K with room left for KV cache. That is the quant a 24 GB 4090 cannot hold. On 24 GB cards the sparse Gemma 4 26B-A4B (25.2B total, 3.8B active, ~16.9 GB at Q4_K_M) is the Gemma pick instead.

RTX 5090 vs RTX 4090 — is the upgrade worth it for LLMs?

Yes if you want to step past 24GB. The 5090 unlocks Qwen 3.6 27B at Q8 quality, Qwen 3.6 35B-A3B at Q6, or DeepSeek V3 at heavy quants. On 24GB-and-under workloads, the 5090 is ~60% faster than the 4090 (1792 vs 1008 GB/s bandwidth) but the cost premium is steep. Buy the 5090 for the VRAM ceiling, not just the speed.

Can the RTX 5090 run a 70B model?

Yes at degraded quants. Llama 3.3 70B at Q3_K_S uses about 28 GB — fits the 5090 with KV cache for short context. Quality is degraded vs Q4 but workable. For full Q4 70B you still need 48 GB+, so either two 5090s or a workstation GPU.

← All guides

Hardware July 24, 2026

Best Local LLM for RTX 5090 (July 2026): 32GB VRAM Picks + OpenClaw Setup

The RTX 5090 jumped the consumer LLM ceiling from 24 GB to 32 GB VRAM and nearly doubled memory bandwidth (1008 → 1792 GB/s) over the RTX 4090. That's enough headroom to run Qwen 3.6 27B at Q8 (near-FP16) with 64K context, or step up to MoE models with 35B+ parameters.

Just bought an RTX 5090?

See our AI training options. We'll set up OpenClaw + Ollama to run all your AI locally on the 5090, free.

🎮 THE RTX 5090 — 32 GB FOR LOCAL LLMs

The RTX 5090's 32 GB is the new-card pick for local LLMs — more headroom than a 24 GB 4090/3090 for bigger context and models. Need 70B at long context? Step up to the 96 GB Blackwell.

5090GIGABYTE RTX 5090 32 GB ↗ 4090GIGABYTE RTX 4090 24 GB ↗ 96GBRTX PRO 6000 Blackwell 96 GB ↗

Updated August 8, 2026: Gemma 4 31B added — the 5090’s 32 GB is the first consumer card that holds the dense Gemma at Q6_K (~25 GB), a quant 24 GB cards can’t fit.

Bottom Line

Best overall pick: Qwen 3.6 35B-A3B (MoE) at Q6_K — ~80 tok/sec, 35B-class quality
Best dense model / 256K context: Gemma 4 31B at Q6_K (~25 GB, multimodal)
Best agentic coding: Laguna XS 2.1 at Q4_K_M (20 GB — 33B MoE, 3B active, 70.9% SWE-Bench Verified)
Best for OpenClaw production: gpt-oss 20B at Q8_0 (cleanest tool calls)
Best premium 27B: Qwen 3.6 27B at Q8_0 (near-FP16)
Best squeeze for 70B: Llama 3.3 70B at Q3_K_S (fits, but quality compromised)

Top Picks for RTX 5090 (32 GB VRAM, 1792 GB/s bandwidth)

1. Qwen 3.6 35B-A3B (Q6_K) — best overall

Mixture-of-Experts variant of Qwen 3.6 (April 22, 2026). 35B total params, 3B active per token. At Q6_K uses about 28 GB. The 5090’s bandwidth + MoE design = blistering inference.

ollama pull qwen3.6:35b-q6_K
openclaw config set agents.defaults.models.chat ollama/qwen3.6:35b-q6_K

Expected speed: 75-90 tokens/sec.

2. Laguna XS 2.1 (Q4_K_M) — best agentic coding

Poolside’s July 2, 2026 release. 33B total params, 3B active per token (MoE), 256K native context, 70.9% on SWE-Bench Verified. laguna-xs-2.1:q4_K_M is 20 GB — on a 24 GB card you must cap context at 8-16K, but the 5090’s 32 GB leaves ~12 GB for KV cache, so you can run roughly 64K context. Skip Q8: those weights are ~33 GB and do not fit.

ollama pull laguna-xs-2.1
openclaw config set agents.defaults.models.chat ollama/laguna-xs-2.1

Expected speed: 45-60 tokens/sec (3B active params + 1792 GB/s bandwidth).

3. Gemma 4 31B (Q6_K) — best dense model, 256K context

Google’s dense Gemma 4, 30.7B params across 60 layers with a 256K context window and multimodal text + image input. Q4_K_M is 18.3 GB and Q6_K is 25.2 GB, so the 5090 is the first consumer card that holds the near-lossless quant with KV cache left over — a 24 GB 4090 has to drop to Q4 or run the sparse 26B-A4B instead. Native structured tool use makes it usable for agent loops.

ollama pull gemma4:31b-q6_K
openclaw config set agents.defaults.models.chat ollama/gemma4:31b-q6_K

Expected speed: 45-55 tokens/sec.

4. gpt-oss 20B (Q8_0) — best for OpenClaw production

OpenAI’s 20B at full Q8 uses about 22 GB. Cleanest tool-call JSON of any open-weight model.

ollama pull gpt-oss:20b-q8_0
openclaw config set agents.defaults.models.chat ollama/gpt-oss:20b-q8_0
openclaw run --agent --max-hours 8 "Implement the spec end-to-end"

5. Qwen 3.6 27B (Q8_0) — premium quality

Full Q8 of the April 22 release uses about 30 GB with 32K context. Near-FP16 quality. Speed: ~45 tok/sec.

6. Mistral Small 4 (119B-A6B MoE, IQ3_XS) — premium reasoning squeeze

Mistral’s March 16, 2026 release. 119B total params, 6B active. At IQ3_XS uses about 30 GB. Quality is degraded at IQ3 but the underlying model is premium tier.

7. Qwen 3.5 122B-A10B (IQ2_XXS) — biggest squeeze

For breadth of knowledge over inference quality. ~30 GB at IQ2_XXS. Note: Qwen 3.5 has the Ollama tool-calling bug — pair with gpt-oss for agent loops.

What Fits in 32 GB VRAM (RTX 5090)

Model	Quant	VRAM	Tok/sec
Qwen 3.6 35B-A3B (MoE)	Q6_K	~28 GB	75-90
Gemma 4 31B (dense) ← 256K ctx	Q6_K	~25 GB	45-55
Laguna XS 2.1 (MoE) ← coding	Q4_K_M	~20 GB	45-60
Qwen 3.6 27B	Q8_0	~30 GB	40-50
gpt-oss 20B	Q8_0	~22 GB	70-85
Mistral Small 4 (119B-A6B)	IQ3_XS	~30 GB	50-65 (MoE)
Llama 3.3 70B	Q3_K_S	~28 GB	15-22 (degraded)

OpenClaw Setup on RTX 5090

ollama pull qwen3.6:35b-q6_K
ollama pull gpt-oss:20b-q8_0
openclaw config set agents.defaults.models.chat ollama/qwen3.6:35b-q6_K
openclaw config set agents.defaults.models.agent ollama/gpt-oss:20b-q8_0
openclaw config set agents.defaults.keep_alive 30m

Common Mistakes on RTX 5090

Running Llama 3.3 70B at IQ2 because it fits. Quality at IQ2 is so degraded that Qwen 3.6 27B at Q8 beats it on every benchmark and runs 2-3x faster.
Maxing context to 256K. KV cache at 256K eats 20+ GB. Cap at 64K-128K depending on the model.
Pulling Laguna XS 2.1 at Q8 because you have 32 GB. The Q8 weights are ~33 GB — they do not fit before you count KV cache. Use Q4_K_M (20 GB) and spend the headroom on context instead.
Buying the 5090 just for tokens/sec. The real value is the 32 GB VRAM ceiling. If you only run 24GB-and-under models, the 4090 is half the price and still fast.

🛒 Mac alternative

Want 32GB+ unified memory without the GPU build? Mac Studio Ultra delivers.

MINIMUM · 24 GB

Apple MacBook Pro M-series 24 GB unified — smaller than 5090 but fanless, portable, silent.

Check current price on Amazon → RECOMMENDED · 48 GB+

Premium Mac for 48 GB+ More unified memory than a 5090, full 70B Q4 capable.

Check current price on Amazon →

Amazon affiliate links — we earn a small commission at no cost to you.