Can I Run Qwen3.8 Flash Next Locally (2026): Yes, on 128GB

Bottom Line

• Yes, you can run it locally, but only on a 128GB box. The smallest GGUF, UD-IQ1_S, is 67.56 GiB. The usual 4-bit pick, UD-IQ4_XS, is 87.25 GiB.
• “6B active” does not mean 6B of memory. The model has 125B MoE parameters, a 51B n-gram embedding table and a 4B MTP head. You must store all of it.
• llama.cpp supports it. PR #27742 was merged on 27 August 2026.
• It is fast once loaded. Only 6B parameters are read per token. Community reports show 16.8 tok/s plain and up to 47.1 tok/s with speculation on Strix Halo.
• The fact most guides miss: “4-bit” is two different answers at 96GB. UD-IQ4_XS fits in 96GB. UD-Q4_K_XL (103.69 GiB) does not.

What the Model Is

All figures below come from the Qwen model card and the Hugging Face API, read on 17 September 2026.

Qwen calls it an experimental preview of the Qwen4 architecture. The model card says the n-gram embedding is “more amenable to offloading than Mixture-of-Experts”. That matters for the hardware choice below.

Every Published Quant, Measured

Sizes come from the Hugging Face file listing for unsloth/Qwen3.8-Flash-Next-GGUF, read on 17 September 2026. We summed multi-part files.

The same repository also has an MTP file set of 22.93 GiB and a vision projector of about 0.85 GiB. Add the projector if you want image input.

A 24GB or 32GB GPU cannot hold any of these. Even UD-IQ1_S is more than twice the memory of a 32GB card.

Why 6B Active Still Needs 128GB

Active parameters set speed. Total parameters set memory. Most people read only the first number.

Per token, the model reads about 6B parameters. That is why decode is fast on a 128GB box with modest bandwidth. But every expert and every row of the n-gram table must sit somewhere the runtime can reach.

The n-gram table is the part that surprises people. Saiyam Pathak’s Kubesimplify write-up (27 August 2026) says the table expands to about 102 GB at full precision. The GGUF quantizes it, and the same write-up reports about 19% higher perplexity for the UD-IQ1_S build as a result.

One community recipe keeps that table off the GPU. On the NVIDIA developer forum, maci.stgn pins it to CPU with -ot "per_layer_token_embd.weight=CPU" and streams it from NVMe. That lets UD-Q4_K_XL run on a single DGX Spark.

Runtime Support, Checked Today

We searched the llama.cpp repository through the GitHub API on 17 September 2026.

So a current llama.cpp build loads the model. The MTP speedups in the table below need community branches today. We did not check Ollama support, so do not assume it.

Community Speed Numbers

We did not measure these. Each row names its source. Treat them as single-user reports.

The Vulkan PR #28501 also reports Strix Halo prefill at 426 → 507 tok/s for an 8K prompt on a Q5_K quant.

One warning from discussion #27950 is worth repeating. An earlier MTP port showed good tok/s while it produced noise above about 1K prompt tokens. Read the output, not only the counter.

To measure your own box, run llama-bench -m <file>.gguf -p 512 -n 128. Then test a real prompt in llama-server at the context length you plan to use.

What to Buy

If you already own a 128GB Strix Halo box or a DGX Spark, download UD-IQ4_XS and try it this week. That is the setup most community reports use.

If you are buying for this model, the two 128GB options are these. Prices are as of September 2026 and move weekly.

On the AMD side, the Strix Halo 128GB mini PC is the box behind most of the speed numbers above. On the NVIDIA side, the DGX Spark gives you CUDA and the forum recipes.

A 96GB GPU is a narrow fit. An RTX PRO 6000 costs about $15,600-16,000 as of September 2026. It holds UD-IQ4_XS with little context room, and it cannot hold UD-Q4_K_XL. For this model, a 128GB box costs much less.

For the general tier question, see the local LLM estimator or the gear page.

Honest Caveats

1. We did not measure tok/s. Every speed above is a named community report, on patched or branch builds.
2. Quant quality is not well measured. The 19% perplexity figure is one report on one quant. We found no KL-divergence table for these files.
3. We did not check Ollama. Use llama.cpp or Unsloth Desktop, which the Unsloth README names.
4. Sizes are as of 17 September 2026. Unsloth re-uploads quants when recipes improve.
5. The model is a preview. Qwen calls it experimental, and the license is qwen-community-1.0, not Apache 2.0. Read it before commercial use.

Sources

• Qwen/Qwen3.8-Flash-Next model card — parameters, experts, context, license
• Hugging Face API: Qwen/Qwen3.8-Flash-Next — creation date, stored parameter count
• Hugging Face API: unsloth/Qwen3.8-Flash-Next-GGUF file tree — every quant size
• llama.cpp PR #27742 — architecture support, merged 27 August 2026
• llama.cpp discussion #27950 — Strix Halo speed reports
• llama.cpp PR #28501 — Vulkan 512-expert fix and prefill numbers
• Kubesimplify: Running Qwen3.8-Flash-Next on a DGX Spark — DGX Spark UD-IQ1_S speed, n-gram table size, perplexity cost
• NVIDIA developer forum: UD-Q4_K_XL on DGX Spark — NVMe streaming recipe

See Also

• Qwen3.8 Flash Next 2-bit on 64GB or 96GB — the GSQ-RCO GGUFs, and why their n-gram shard can stay on disk
• Can I run GLM-5.3-Flash locally? — the same 128GB question for a 321B MoE, where llama.cpp support is not merged yet
• DGX Spark vs Strix Halo for local LLMs — how to choose between the two 128GB boxes
• Which Strix Halo mini PC to buy — the AMD options and their current prices
• Best local LLM for 128GB VRAM — what else fits at this tier
• Is 96GB of VRAM enough in 2026? — why the 96GB tier is a narrow fit for new MoE models
• IQ4_XS vs Q4_K_M vs Q4_K_XL — why two “4-bit” files differ by 16 GiB here
• Qwen3.8 27B on an RTX 3090 — the Qwen3.8 model that fits a 24GB card