Best Local LLM for 96GB VRAM (2026): Qwen3.5 122B Wins

Bottom Line (as of September 2026)

• Best model for 96GB of VRAM: Qwen3.5-122B-A10B at UD-Q4_K_XL. The file is 77.03 GB (71.74 GiB). That leaves about 24 GiB for context. It is Apache-2.0, and its model card scores 72.0 on SWE-bench Verified against 62.0 for gpt-oss 120B in the same table.
• Fastest model: gpt-oss 120B. Its native MXFP4 GGUF is 63.39 GB (59.03 GiB). Hardware Corner measured 148.17 tok/s on one RTX PRO 6000 with llama.cpp at 12K context, and 83.03 tok/s at 131K.
• 96GB of VRAM is one product. The RTX PRO 6000 Blackwell is the only card with 96GB on one die. NVIDIA lists it at $16,000. Central Computer had it at $14,999.99, in stock on 2026-09-30.
• It has no bandwidth advantage over an RTX 5090. Both move 1,792 GB/s. You pay about 3.5x for capacity, not speed.
• The reason to want 96GB over 48GB: Llama 3.3 70B at Q8_0 is 74.98 GB and fits on one card. At 48GB a 70B is a Q4 model.
• What still does not fit: any Qwen at 235B or larger. Qwen3-235B-A22B at IQ4_XS is 125.48 GB.
• The Max-Q is not cheaper. Same 96GB, 300W instead of 600W, and the same $14,999.99 at Central Computer. Buy it for power, not price.

Ready to buy? See the tested hardware list with current prices.

Looking for 96GB of system RAM, not VRAM? That is a different machine and a different model list — go to best local LLMs for 96GB RAM instead.

First, Make Sure You Mean VRAM

These two searches land on the same page everywhere on the web, and they are not the same question.

96GB of system RAM describes a Mac Studio, a 96GB DDR5 workstation, or a unified-memory box. 96GB of VRAM describes one $16,000 card, or a rack of smaller ones wired together.

The model list barely changes between them. What changes is how fast the tokens come out:

On a dense model, generation speed tracks that column closely, because every token requires reading the whole weight set out of memory. A 70B at Q8 that fits in both a 96GB Mac Studio M5 Ultra and a 96GB PRO 6000 will run about 1.5x faster on the card (1,792 against 1,200), and about 6.5x faster than on a DGX Spark. These are bandwidth ratios, not measurements.

That is the honest case for VRAM at this tier. It is not that more fits. It is that the same thing runs faster.

The comparison most guides miss: Apple’s 96GB M5 Ultra Mac Studio configuration cost $5,499 on apple.com on 2026-09-21. That is about a third of the card’s price for 67% of its bandwidth, in a whole computer. The card wins on CUDA and on prompt processing. On generation speed per dollar, it loses.

The Number Nobody Puts in the Comparison

Here is the spec that should change how you think about this card.

The RTX PRO 6000 Blackwell’s memory bandwidth, from NVIDIA’s own product page, is 1,792 GB/s. The RTX 5090’s memory bandwidth, from its 512-bit GDDR7 interface at 28 Gbps, is 1,792 GB/s.

They are the same number.

The PRO 6000 has more CUDA cores — 24,064 against the 5090’s 21,760 — so it is modestly faster on compute-bound work like prefill and fine-tuning. But LLM token generation is memory-bandwidth-bound, not compute-bound. On a model that fits in 32GB, a 5090 will generate tokens at essentially the same rate as a card costing three times as much.

So the purchase decision reduces to one question, and it is a capacity question:

Does your model fit in 32GB? If yes, buy a 5090 and keep about $10,700. If no, and it fits in 96GB, this card is the only single-die way to do it.

Nothing about “the professional card is faster” survives contact with the spec sheet. It holds more. That is the product.

(Both figures sit in the spec table on our own RTX PRO 6000 vs RTX 5090 page. What almost no buying guide does is draw the conclusion from them.)

What 96GB of VRAM Actually Runs

Sizes below are byte-exact Hugging Face file sizes, read 2026-09-30. Weights only — add the KV cache on top. Card “GB” is close to GiB: the 48GB RTX PRO 5000 reports 47.8 GiB in nvidia-smi, so expect a little under 96 GiB here. Read the GiB column.

Read the Qwen3-235B rows together and you get the ceiling. A 235B MoE at 2-bit fits your 96GB card and a 235B MoE at any quantization worth running does not. The newer Qwen3.5-397B-A17B is worse: its smallest file (UD-IQ1_M) is 106.82 GB. 96GB is not “almost 128GB” — it lands exactly below the frontier open-weight tier, which is why the honest picks at 96GB are a 120B MoE and a 70B at high precision, not a 235B at a bad one.

The Pick: Qwen3.5-122B-A10B at Q4

If you own 96GB of VRAM and want the strongest model that fits, run this one.

It is a 122B mixture-of-experts with 10B active parameters per token. At UD-Q4_K_XL it is 71.74 GiB, so it leaves about 24 GiB for context. Its model card scores 72.0 on SWE-bench Verified, 72.2 on BFCL-V4 and 79.5 on TAU2. The same table gives gpt-oss 120B 62.0 on SWE-bench Verified. These are vendor-reported scores. We found no measured tok/s for it on this card, so treat speed as unknown until you run it.

Run gpt-oss 120B instead if speed or context matters more. It ships in native MXFP4 at 59.03 GiB, so you make no quantization compromise. It leaves about 35 GiB free, and it was measured at 148.17 tok/s on this card (Hardware Corner, llama.cpp build 6112, flash attention on, 12K context). Its tool-call output is the cleanest we have found for OpenClaw agent loops.

For coding agents, look at Laguna S 2.1. Poolside’s 118B-A8B fits at UD-Q4_K_XL (68.35 GiB) with a 1M-token window and scores 70.2% on Terminal-Bench 2.1 per its card. Poolside’s own Q4_K_M is 96.03 GB and does not fit.

The other model worth knowing about is Llama 3.3 70B at Q8_0. At 74.98 GB it is the demonstration of what this card buys: a 70B running at 8-bit, entirely resident, on one device. On a 48GB setup the same model runs at Q4. Whether that quality difference is worth $16,000 is a question we take up honestly below — the published evidence on quantization suggests it is smaller than most people assume.

What to Buy

The only 96GB card — RTX PRO 6000 Blackwell

96GB of GDDR7 with ECC, 1,792 GB/s, 24,064 CUDA cores. NVIDIA’s list price is $16,000 as of September 2026, up 87% from its $8,565 launch — GDDR7 supply is the reason, the same shortage that put the RTX 5090 above $4,300. On 2026-09-30 Central Computer listed the PNY Workstation Edition at $14,999.99, in stock. Newegg listings started at $15,599 on 2026-09-12. Our card-versus-card comparison goes deeper on batched serving and decode speed.

96GBRTX PRO 6000 Blackwell 96 GB ↗

96GB 300WRTX PRO 6000 Blackwell Max-Q 96 GB ↗

Check the variant before you pay. There are two: the Workstation Edition at 600W (first link above), and the Max-Q at 300W (second link) with a blower cooler, 24,064 CUDA cores and the same 96GB at essentially the same bandwidth. For a machine doing inference around the clock, half the power for the same memory is the better buy. Do not expect a discount for it: on 2026-09-30 Central Computer listed the PNY Max-Q at the same $14,999.99 as the Workstation Edition, and Newegg listed it at $17,999.99. The “$8,299 Max-Q” deals that still circulate are from 2025. Listings are not always explicit about which one they are selling — this is the same trap we flagged on the 48GB RTX PRO 5000, which ships in 48GB and 72GB versions under one name.

The cheaper answer, if 32GB is enough — RTX 5090

Same 1,792 GB/s, under a third of the price: the September 2026 floor was $4,299 (Walmart and Micro Center), with most listings higher. If the models you actually run fit in 32GB — anything up to a 30B-class model at Q4 with a long context, or a 70B at low quant with offload — you are buying identical token throughput for about $10,700 less.

32GBGIGABYTE RTX 5090 WINDFORCE 32 GB ↗

96GB across three cards — the build that looks cheaper and is not

Three RTX 5090s total 96GB and cost about $12,900 at the September 2026 floor of $4,299 each, if you can find three in stock. It is a worse machine for this job:

• The memory is three separate 32GB pools. A 72-75GB model must be sharded, and every token crosses PCIe.
• 1,725W of GPU power before the rest of the system. See what PSU a local AI rig needs.
• Most consumer boards cannot give three cards meaningful CPU-connected lanes. Our multi-GPU motherboard guide found that both mainstream B650 boards top out at a single x16 slot.

Two used RTX 3090s at 48GB remain the sane budget route to large-model work, and we compared the tradeoff in detail in dual 3090 vs 5090.

24GBEVGA RTX 3090 24 GB ↗

Same model list, a quarter of the price — a 128GB unified box

If the goal is “run gpt-oss 120B locally” and not “run it fast”, a DGX Spark at $4,699 or an ASUS Ascent GX10 ($6,999 on the ASUS US store in September 2026) holds 128GB of unified memory and runs CUDA. The catch is in the bandwidth table above: 273 GB/s against the card’s 1,792 GB/s. The same weights, roughly a sixth of the token rate on dense models.

128GBNVIDIA DGX Spark 128 GB ↗

128GBASUS Ascent GX10 128 GB ↗

The Honest Caveat

Most people who search for 96GB of VRAM should not buy it.

The case for the card is narrow and real: you need a 122B-class MoE at 4-bit or a 70B at 8-bit, resident on one device, at full bandwidth, with CUDA, and the machine earns money. Research labs, small teams serving an internal model, and people fine-tuning rather than only inferencing.

Outside that, the arithmetic is unkind. The gap between a 70B at Q4 and the same model at Q8 is small — Red Hat’s half-million-evaluation study found 4-bit recovering 98.9% of full-precision accuracy on HumanEval, and 8-bit 99.9%. You are paying about $10,700 over a 5090 to close a one-point gap, on a card that generates tokens no faster.

Buy 96GB because a model you need genuinely does not fit in 32GB. That is the whole justification, and it is enough on its own — it just is not the justification most buying guides give.

Before you order parts, check the tested hardware list for current prices by tier.

See Also

• How Much VRAM Does gpt-oss 120B Need at Full 128K Context? — the context arithmetic for the model this tier is built around
• Best Local LLMs for 96GB RAM — the system-memory version of this question
• Best Local LLM for 128GB of VRAM — the tier above, and the trap inside the query
• RTX PRO 6000 vs RTX 5090 — the same two cards, head to head
• 64GB of VRAM Is Not 64GB of Unified RAM — the same distinction, one tier down
• Best 48GB VRAM Setup — the tier below, four ways
• Is the DGX Spark Worth It? — the 128GB unified alternative, priced out
• DGX Spark Price: Every GB10 Box Compared — the $4,699 list price, its February increase, and every OEM GB10 box beside it
• Dual RTX 3090 vs RTX 5090 — VRAM capacity against bandwidth
• Motherboard and CPU for a Multi-GPU Rig — why three cards is harder than it looks
• NVIDIA Price Hikes: What to Buy — why every number here moved in 2026
• RTX PRO 6000 Max-Q vs Workstation Edition — same 96GB and same 1,792 GB/s, half the power: picking the variant
• Is 96GB of VRAM Enough in 2026? — whether the tier suffices, with the context arithmetic
• MIG on a Workstation GPU — splitting this card into 4 x 24GB isolated instances, and what it costs