Qwen3.8 Flash Next 2-Bit (2026): 64GB Works, 96GB Is Better

Bottom Line

• The full file does not fit a 64GB machine. GSQ-RCO Q2_0 is 61.86 GiB and IQ2_XS is 63.35 GiB. With an OS and a KV cache, 64GB is not enough to hold all of it.
• But you do not have to hold all of it. Shard 2 is the 26.82 GiB n-gram table. It is identical in all three builds, and ISTA-DASLab says it can stay memory-mapped on SSD. Then only shard 1 must be resident: 35.04 GiB (Q2_0) or 36.53 GiB (IQ2_XS).
• So 64GB works, with a cost. The table is read from disk. On an AMD iGPU, llama.cpp turned this mode off by default because it halved prefill speed (issue #28160).
• 96GB-class memory is the clean tier. A 128GB Strix Halo box or a DGX Spark holds the whole file in memory with room for long context.
• The 2-bit GSQ-RCO file is smaller than Unsloth’s 1-bit file. IQ2_XS is 63.35 GiB. Unsloth’s UD-IQ1_S is 67.56 GiB.

The Files, Measured

We read these sizes from the Hugging Face API on 20 September 2026. The repository was created on 7 September 2026 and last changed on 18 September 2026. The license is Apache 2.0.

The GiB totals are our sums of the two shards. The GB column is from the model card. The bpw column is the average over the transformer weights, not over the whole file.

Why Shard 2 Is Always 26.82 GiB

All three second shards are exactly 28,800,138,432 bytes. That is not a coincidence.

The model card says shard 2 holds only the per-layer n-gram embedding table (per_layer_token_embd, 51.2B parameters). It is stored at IQ4_NL, a fixed 4.5 bits per weight, and it is excluded from the RCO search.

Our arithmetic: 51.2B x 4.5 bits / 8 = 28.8 GB. That matches the file.

The repository also ships tensor allocation files for shard 1 of each build. They confirm the split. Each shard-1 file lists 1,223 tensors, and none of them is the n-gram table. The per-tensor types differ a lot between builds:

Counts exclude the BF16, F16 and F32 tensors that every build keeps. The model card also says ffn_down_exps is held at Q2_0 in all builds, because its 640 rows do not divide by 256.

What this means for you: the “2-bit” label only describes shard 1. Shard 2 is a 4.5-bit table in every build. That is why the 2-bit and 3-bit downloads differ by only 8.77 GiB (our subtraction).

Does It Fit 64GB?

Plainly: the whole file does not. 61.86 GiB of weights on a 64GB machine leaves almost nothing for the OS, the KV cache and the runtime.

Shard 1 does. The model card says to run with -lm mmap --lazy-mode on, keep shard 2 on an SSD, and plan for only shard 1 in GPU memory. The n-gram table is a lookup: the model reads one row per token, not the whole table.

On a 64GB Mac, the GPU cannot use all 64GB by default. macOS keeps part of it for the system. Check that your shard 1 plus context fits under that limit before you download.

The iGPU catch

This is the part that matters for 64GB Strix Halo buyers.

llama.cpp added the lazy loading flag in PR #27969 (merged 30 August 2026). Then a Strix Halo user found that lazy loading halved prefill for this model on Vulkan: pp512 fell from about 429 to 216 tok/s (#28160). PR #28326 (merged 8 September 2026) fixed it by turning lazy loading off by default on iGPUs.

So on a 64GB iGPU machine, you must turn the mode on yourself, and you accept the slower prefill that the default now avoids. PR #28326 also renamed the modes (auto, large, all, off). Run llama-cli --help on your build to see which values it accepts, because the model card still shows on.

On a 128GB box you can leave the table in memory and skip the problem.

KV Cache Headroom

The Qwen config.json lists 48 layers, and a full-attention layer every 4th layer. The other 36 are linear-attention layers with a fixed-size state. Full-attention layers have 2 KV heads with a head size of 256.

Our estimate, at 16-bit KV cache for the 12 full-attention layers only:

Math: 12 layers x 2 (K and V) x 2 heads x 256 x 2 bytes = 24 KiB per token. This ignores the linear-attention state, the indexer and runtime buffers, so treat it as a floor. It is small next to the weights. The weights set the tier, not the context.

Quality and Speed, From the Model Card

These are ISTA-DASLab’s numbers, not ours. Task average is the mean of AIME25, GPQA-Diamond and LiveCodeBench v6, at xhigh reasoning effort.

The card warns that at lower reasoning effort, the gap to BF16 can be larger.

On speed, the card reports Q2_0 at 367.49 prompt tok/s and 93.79 decode tok/s, against 108.19 and 70.30 for IQ2_XS, over 55 prompts in llama.cpp. The card does not name the hardware, so do not compare these numbers to your machine. The ratio is the useful part: at 2 bits, pick Q2_0 unless you need the small LiveCodeBench lead of IQ2_XS.

Runtime Support, Checked Today

We checked these on GitHub on 20 September 2026.

Use a current build. If prefill is very slow, check issue #28355 before you blame the quant.

What to Buy

If you own a 64GB machine, download Q2_0, keep shard 2 on a fast NVMe drive, and turn lazy loading on. Measure prefill before you decide it is good enough.

If you are buying for this model, buy 128GB. That holds the whole IQ3_XXS file, which is the build ISTA-DASLab recommends, and it lets you keep the table in memory. Prices are as of September 2026 and move weekly.

On the AMD side, a Strix Halo 128GB mini PC holds the full file with room for context. On the NVIDIA side, the DGX Spark gives you CUDA.

Do not buy a 64GB machine for this model. It works, but you pay a disk-read cost on every prompt, and the 3-bit build that matches BF16 on AIME25 is tight. Apple discontinued the M4 Max and M3 Ultra Mac Studio on 25 August 2026, so do not plan around those.

For the general tier question, see the local LLM estimator or the gear page.

Honest Caveats

1. We did not run these files. Every speed and quality number is from the model card or a named GitHub issue.
2. We did not measure the lazy-mode cost on a 64GB machine. The halved-prefill number is for an 87 GiB Unsloth build on a 128GB Strix Halo, not for GSQ-RCO on 64GB.
3. The KV numbers are our estimate from config.json. Real use adds runtime buffers.
4. Mode names changed. The model card says --lazy-mode on. Check your build’s --help.
5. Sizes are as of 20 September 2026. The repository changed on 18 September 2026 and may change again.

Sources

• Hugging Face API: ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF — shard sizes, dates, license
• ISTA-DASLab model card — shard 2 contents, memory advice, quality and speed tables
• Tensor allocation files in the same repository (tensor-allocation/) — per-tensor quant types
• Hugging Face API: unsloth/Qwen3.8-Flash-Next-GGUF tree — UD-IQ1_S size for comparison
• Qwen/Qwen3.8-Flash-Next config.json — layers, KV heads, head size
• llama.cpp #27742, #27969, #28160, #28326, #28355 — support and lazy-mode history

See Also

• Can I run Qwen3.8 Flash Next locally? — every Unsloth quant size and community speed numbers
• Best local LLM for 64GB RAM — what fits 64GB without disk tricks
• Best Local LLMs for 96GB RAM — the tier that holds this file in memory
• DGX Spark vs Strix Halo for local LLMs — how to choose between the two 128GB boxes
• Which Strix Halo mini PC to buy — the AMD options and their current prices
• Is 96GB of VRAM enough in 2026? — why 96GB is a narrow fit for new MoE models