Best 20B to 35B Local LLMs (August 2026): The Band That Fits One GPU

Bottom Line (August 2026)

• Best overall — Qwen 3.6 27B, dense, Q4 on a 24GB card or Q6_K on a 32GB Mac. 262K context, 77.2 SWE-bench Verified.
• Highest score in the band — Gemma 4 31B, dense, 24GB at Q4. Tops BenchLM’s August aggregate at 60.09 and handles images. It fills a 24GB card, so budget your context.
• Best on 16GB — gpt-oss 20B, about 12GB at Q4, and the cleanest tool-call JSON of anything here.
• Fastest — Nemotron 3 Nano 30B-A3B or Qwen 3.6 35B-A3B. Both activate ~3B parameters per token.
• Best coding agent — Laguna XS 2.1 (33B/3B MoE), 256K context, 63.1% SWE-bench Multilingual, INT4 and NVFP4 checkpoints.

This band exists because 24GB is the reference consumer GPU in 2026 and 32GB is the common Mac. A 27B dense model at Q4 lands almost exactly on that budget. Everything below is organized by what your hardware actually holds.

The Band at a Glance

Dense vs MoE in This Band

The 20B-35B range now splits cleanly in two, and the split matters more than any single benchmark.

Dense models — Qwen 3.6 27B, Gemma 4 31B, Devstral Small 24B — run every parameter for every token. You pay full memory bandwidth per token, so decode is slower, but each answer gets the model’s full depth. These win on hard single-pass reasoning.

MoE models — Qwen 3.6 35B-A3B, Nemotron 3 Nano 30B-A3B, Laguna XS 2.1, Gemma 4 26B-A4B — hold all weights in memory but activate roughly 3-4B per token. Memory cost stays high; bandwidth cost collapses. NVIDIA measured Nemotron 3 Nano at 2.2x the throughput of gpt-oss 20B and 3.3x that of Qwen3-30B-A3B-Thinking-2507 on an 8K-in / 16K-out workload.

The practical rule: if you are watching a token stream and waiting, take the MoE. If you are asking one hard question and reading the answer carefully, take the dense model.

By Hardware Tier

16GB VRAM (RTX 4060 Ti 16GB, RTX 5060 Ti)

Only the small end of the band fits. gpt-oss 20B at Q4 is the pick — about 12GB, and OpenAI states it runs in 16GB of memory while matching o3-mini on coding, general problem solving, and tool calling. Gemma 4 26B-A4B at Q4_K_M (~15GB) and Devstral Small 24B at Q4_K_M (~14.5GB) also fit, with little left for context.

Do not force a dense 27B onto 16GB. You would need Q3 or lower, and a well-quantized 20B beats a badly quantized 27B every time.

24GB VRAM (RTX 3090, RTX 4090)

This is the tier the whole band is designed around. A 24GB RTX 3090 holds every model here at Q4 with room for a real context window, and it is the cheapest card that does.

• Qwen 3.6 27B Q4 (~18GB) — the default. Leaves ~6GB for KV cache, which is a comfortable 32-64K context.
• Gemma 4 31B Q4 (~24GB) — the highest score in the band, but it fills the card. Keep context short or run it on a 32GB card instead.
• Qwen 3.6 35B-A3B Q4_K_M (~21GB) — same memory class, much faster decode.
• Laguna XS 2.1 INT4 (~20GB) — the agentic coding pick, with DFlash speculator weights that roughly double decode speed.

32GB VRAM (RTX 5090)

Everything in the band fits at a higher quant. Run Qwen 3.6 27B at Q6_K or Q8_0 instead of Q4, or run Gemma 4 31B at Q4 with a full-length context and no memory anxiety. On NVFP4-capable Blackwell cards, the NVFP4 checkpoints of Laguna XS 2.1 and Nemotron 3 Nano hold near-FP8 quality at half the memory.

32GB unified memory Macs

Same models, slower decode, no CUDA. Qwen 3.6 27B at Q6_K uses about 22GB and is the daily driver. gpt-oss 20B at Q8_0 (~24GB) is the agent model. Qwen 3.6 35B-A3B at Q5_K_M (~26GB) fits but leaves little headroom.

On Apple Silicon, MLX runs 2-3x faster than llama.cpp for token generation. On a 32GB M4 Pro, a 30B-A3B MoE hits 130 tok/sec in MLX versus 43 tok/sec in Ollama. Use MLX for standalone speed, Ollama when you need OpenClaw’s tool-calling integration. Full detail in Best Local LLM for 32GB RAM.

48GB unified memory Macs

The band stops being a constraint. Run Gemma 4 31B and gpt-oss 20B loaded simultaneously — one for reasoning, one for tool calls — or run any single model at Q8 with a 128K context.

The Models, Briefly

Qwen 3.6 27B — released April 22, 2026. Dense, 262K context, 77.2 on SWE-bench Verified, which beats the far larger Qwen 3.5 MoE on agentic coding. About 18GB at Q4, 22GB at Q6_K, 30GB at Q8_0. If you only take one model from this page, take this one.

Gemma 4 31B — Google’s dense multimodal entry. BenchLM’s August 2026 aggregate puts it at 60.09, the highest score in the band and above Qwen 3.6 27B’s 52.83. It reads images. The cost is memory: 24GB at Q4 leaves nothing on a 24GB card.

gpt-oss 20B — OpenAI’s open-weight MoE. Matches o3-mini on coding, general problem solving, and tool calling. The reason to run it is not the benchmark, it is the tool-call JSON, which stays valid across long unattended agent sessions where larger models drift.

Qwen 3.6 35B-A3B — 35B total, 3B active. Roughly 21GB at Q4_K_M, 26GB at Q5_K_M. The speed variant of the Qwen 3.6 family, and the model most people with 32GB can run as an everyday assistant.

Nemotron 3 Nano 30B-A3B — NVIDIA’s hybrid Mamba-Transformer MoE, ~31.6B total with ~3.2-3.6B active per forward pass. BF16 needs about 60GB, so quantize: INT8 and INT4 builds run on 20-32GB cards, and 24GB is enough for inference. Cloud deployments measure 93.7 tok/sec median output. Strong on structured output and JSON.

Laguna XS 2.1 — Poolside’s 33B/3B MoE, built specifically for agentic coding and long-horizon work on a local machine. 256K context, FP8 KV cache, mixed sliding-window and global attention. 63.1% on SWE-bench Multilingual, up 5.4 points from XS.2. Ships in BF16, FP8, INT4, and NVFP4, under the permissive OpenMDW-1.1 license, with GGUF planned for llama.cpp. The open-weight DFlash speculator models roughly double inference speed.

Gemma 4 26B-A4B — 26B total, ~4B active, about 15GB at Q4_K_M and 45+ tok/sec on an M4 Pro. Apache 2.0. The smallest capable footprint here, and the right choice for a second model loaded beside a bigger one.

Devstral Small 24B — Mistral’s dense coding model, ~14.5GB at Q4_K_M. Built for agentic coding workflows. Pair it with a general model rather than running it alone.

What to Avoid in This Band

1. Q3 and below on dense 27B-31B models. Quality drops below what a well-quantized 20B gives you at the same memory.
2. Maxing context on a full card. Gemma 4 31B Q4 on a 24GB GPU with a 128K context will not hold. KV cache is not free.
3. Assuming MoE means less memory. A 35B-A3B model still stores 35B parameters. Only bandwidth drops.
4. Old-generation picks. Qwen 3.5 27B has a tool-calling bug in Ollama (issue #14493). Always take 3.6.

See Also

• Best Local LLM for 32GB RAM — the RAM-tier view of these same models
• Best Local LLM for RTX 3090 — the cheapest 24GB card for this band
• Best Local LLM for RTX 5090 — 32GB VRAM, higher quants
• Best Models for Popular RTX GPUs — per-card picks across every tier
• Best Local Models for OpenClaw — model-first guide for agent runs