Can I Run Ling-3.1-flash Locally? (2026): Yes, on 200GB+
inclusionAI put Ling-3.1-flash on Hugging Face on 13 October 2026. It is a 560B mixture-of-experts model with 25B active parameters per token. We summed every GGUF file in bytes from the Hugging Face API, did the cache arithmetic from config.json, and checked llama.cpp, MLX and Ollama support. The short answer: you need about 200GB of total memory. No 128GB machine holds even the 1-bit llama.cpp file.
Sizing a box for a 560B model?
Use the local model calculator first, then see our AI training options if you want help picking the quant and the hardware before you spend five figures.
Bottom Line
- Yes, but only with about 200GB of total memory. The smallest usable 2-bit GGUF is 159.06 GiB (IQ2_XXS). Q2_K is 204.12 GiB.
- 128GB machines do not run it in llama.cpp. The 1-bit IQ1_S file is 122.01 GiB. That is over a 128GB Mac’s 107.52 GiB default Metal limit and over a DGX Spark’s 121.63 GiB total.
- The 256GB Mac Studio M5 Ultra is the simplest box. It costs $9,499 (apple.com, 22 October 2026) and holds Q2_K with the full 262K context.
- The 512GB Mac Studio cannot be bought yet. Apple’s store still says the option is “coming late October”. Q4_K_M (333.39 GiB) needs it.
- On a PC, use a GPU plus 192-256GB of RAM. Put the always-active trunk on the GPU and the routed experts in system RAM.
- The context is cheap. Only 9 of 72 layers keep a growing cache. The full 262,144-token window costs about 2.8 GiB.
- llama.cpp supports it. Use build b11627 or newer. MLX has code support but no ready quant. Ollama has no library entry.
Ready to buy? See the tested hardware list with current prices.
What the Model Is
All figures come from the Ling-3.1-flash model card, its config.json and the Hugging Face API, read on 22 October 2026.
| Spec | Value |
|---|---|
| Repo | inclusionAI/Ling-3.1-flash |
| Created on Hugging Face | 13 October 2026 |
| Total parameters | 560B (card); 572,120,680,768 in the checkpoint, including the MTP layer |
| Active parameters | 25B per token (card) |
| Layers | 72: 63 Kimi Delta Attention (KDA, linear) + 9 gated MLA |
| Experts | 512 routed + 1 shared, 8 routed active; the first 4 layers are dense |
| Extra layer | 1 multi-token-prediction (MTP) layer for speculative decoding |
| Stored weights | FP8 (E4M3), 128 x 128 blocks |
| Context | Card: “1M-token”. config.json: 262,144 |
| Architecture tag | BailingMoeV3ForCausalLM, model_type bailing_hybrid |
| License | MIT |
The card shows its benchmark results only as images. We quote no scores. The card’s run recipes target servers: SGLang on 8 x H200, or 4 x GB300.
The context claim does not match the config. The card says 1M tokens. config.json sets max_position_embeddings to 262,144 and rope_scaling to null. Both GGUF repos list 262,144. Plan for 262K.
Two-thirds of each token’s reads are fixed. Config.json gives 68 MoE layers with experts of 3 x 4,096 x 1,280 weights. Eight active experts in 68 layers is 8.56B weights. The other 16.4B of the 25B active weights are the trunk: attention, the shared expert, the dense layers and the output head. Every token reads all of it.
Every GGUF Size, in Bytes
We summed the split files in the Hugging Face file trees for bartowski/Ling-3.1-flash-GGUF and AtomicChat/Ling-3.1-flash-GGUF on 22 October 2026. “File bits/weight” is from bartowski’s card.
| File | Source | Bytes | GiB | File bits/weight |
|---|---|---|---|---|
| IQ1_S | bartowski | 131,007,719,744 | 122.01 | 1.83 |
| IQ1_M | bartowski | 145,087,687,008 | 135.12 | 2.03 |
| AD-IQ1_S | AtomicChat | 141,806,435,008 | 132.07 | — |
| IQ2_XXS | bartowski | 170,789,808,544 | 159.06 | 2.39 |
| AD-IQ2_XXS | AtomicChat | 174,144,518,880 | 162.18 | — |
| IQ2_S | bartowski | 188,466,899,360 | 175.52 | 2.64 |
| IQ2_M | bartowski | 213,027,940,928 | 198.40 | 2.98 |
| AD-IQ2_M | AtomicChat | 218,079,853,408 | 203.10 | — |
| Q2_K | bartowski | 219,168,205,376 | 204.12 | 3.06 |
| IQ3_XXS | bartowski | 247,367,821,984 | 230.38 | 3.46 |
| Q3_K_M | bartowski | 275,057,306,272 | 256.17 | 3.85 |
| IQ4_XS | bartowski | 317,077,154,752 | 295.30 | 4.43 |
| Q4_K_M | bartowski | 357,969,521,664 | 333.39 | 5.01 |
| AD-Q4_K_M | AtomicChat | 354,394,733,920 | 330.06 | — |
| Q5_K_M | bartowski | 436,319,120,672 | 406.35 | 6.10 |
| Q6_K | bartowski | 501,582,491,136 | 467.14 | 7.01 |
| Q8_0 | bartowski | 608,380,104,192 | 566.60 | — |
Bartowski marks both 1-bit files “not recommended”. AtomicChat measured its 1-bit files against BF16. AD-IQ1_S picked the same next token 86.41% of the time, and AD-IQ2_M 92.72%. AD-Q4_K_M reached 96.86%.
The MTP layer differs by repo. Bartowski includes it, stored at Q4_0. Add --spec-type draft-mtp to use it. AtomicChat’s card says its files do not include the MTP head.
Downloads shown by the Hugging Face API on 22 October: bartowski 2,172, AtomicChat 1,622, the original FP8 repo 3,292.
Why the Context Costs So Little
Only the 9 MLA layers keep a cache that grows with context. The 63 KDA layers keep a fixed state.
1. The MLA cache. Config.json gives kv_lora_rank 512 and qk_rope_head_dim 64. llama.cpp’s bailingmoe3.cpp caches the compressed latent plus the rope part.
f16: (512 + 64) values x 2 bytes x 9 MLA layers = 10,368 bytes per token
32,768 tokens = 0.32 GiB
131,072 tokens = 1.27 GiB
262,144 tokens = 2.53 GiBAtomicChat’s card gives “about 10 KB per token”, which matches. If a runtime stores V separately from K, the cost rises to 19,584 bytes per token. That is still only 4.78 GiB at 262K.
2. The fixed KDA state. Config.json gives 64 heads with a head dimension of 128 and a convolution kernel of 4.
Recurrent state: 63 layers x 64 heads x 128 x 128 x 4 bytes (F32) = 252 MiB
Conv state: 63 layers x 3 x (3 x 64 x 128) x 4 bytes = 17.7 MiB
Total per conversation ~ 270 MiBThe VQ-GGUF uploader measured 276 MiB of KDA state in their runtime. Our estimate is within 3%.
3. The full budget. Weights + 2.53 GiB cache + 0.27 GiB state + a compute buffer. We budget 2 GiB for the buffer. We did not measure it.
Fit Table by Hardware
Mac rows use the Metal limit. PC rows use VRAM plus system RAM, minus about 8 GiB for the OS. Every row is our arithmetic, not a measurement.
| Hardware | Total memory | Best llama.cpp file | Fits? | Context |
|---|---|---|---|---|
| 128GB Mac (M5 Max) | 107.52 GiB default limit | none | No. IQ1_S is 122.01 GiB | — |
| DGX Spark / GB10 | 121.63 GiB total | none | No in llama.cpp. See the VQ note below | — |
| Strix Halo 128GB | 128GB shared | none | No | — |
| Two DGX Sparks (RPC) | 2 x 121.63 GiB | IQ2_XXS (159.06) | In theory. Untested | Unknown |
| RTX 3090 24GB + 192GB DDR5 | ~216 GiB | IQ2_XXS (159.06) or IQ2_S (175.52) | Yes, with expert offload | 262K |
| RTX 5090 32GB + 256GB DDR5 | ~288 GiB | IQ3_XXS (230.38) or Q3_K_M (256.17) | Yes, with expert offload | 262K |
| Mac Studio M5 Ultra 256GB | 234.38 GiB with the raised limit | Q2_K (204.12) or IQ2_M (198.40) | Yes | 262K |
| RTX PRO 6000 96GB + 256GB DDR5 | ~352 GiB | IQ4_XS (295.30) | Yes, with expert offload | 262K |
| Mac Studio M5 Ultra 512GB | not orderable on 22 Oct | Q4_K_M (333.39) or Q5_K_M (406.35) | Would fit | 262K |
The 256GB Mac needs one command. AtomicChat’s card gives sudo sysctl iogpu.wired_limit_mb=240000 for a 256GB machine. That is 234.38 GiB. Q2_K plus the full cache and state is about 209 GiB. IQ3_XXS (230.38 GiB) leaves under 4 GiB, so skip it.
The one 128GB route is experimental. Baekpica/Ling-3.1-flash-VQ-GGUF stores the experts at about 1.5 bits per weight. All 71 parts total 107.81 GiB. Its card says “ordinary llama.cpp, Ollama and Transformers loaders do not implement this format”. It runs only in the uploader’s ds4-dfm-rs runtime. The card warns about loops and broken characters.
Two DGX Sparks are possible, not proven. NVIDIA says up to four Sparks link over ConnectX-7. llama.cpp can split a model across machines with its RPC backend. Its README calls RPC “proof-of-concept” and “fragile”. We found no report of Ling-3.1-flash running over RPC on two Sparks.
How Fast Will It Be?
We found one measured number, and it is not from llama.cpp. The VQ uploader reports these figures on one DGX Spark, with ds4-dfm-rs (community-reported):
| Workload | Prefill | Decode |
|---|---|---|
| 2K-64K sweep, MTP off | 774.1 tok/s | 17.70 tok/s |
| 2K-32K sweep, two MTP drafts | 861.0 tok/s | 23.58 tok/s |
The card says each token reads about 10.75 GB of weights in that build. A Spark has 273 GB/s, so its ceiling is about 25.4 tok/s. The measured 17.70 tok/s is 70% of that ceiling.
The rows below are bandwidth ceilings, not predictions. Real speed lands below them.
Mac Studio M5 Ultra 256GB, Q2_K (all in unified memory):
25B active x 3.06 bits / 8 = 9.56 GB per token
1,200 GB/s / 9.56 GB = ~125 tok/s ceiling
PC, experts in RAM (trunk on GPU), dual-channel DDR5-4800 = 76.8 GB/s:
IQ2_XXS: 8.56B x 2.39 / 8 = 2.56 GB per token -> ~30 tok/s ceiling
Q2_K: 8.56B x 3.06 / 8 = 3.27 GB per token -> ~23 tok/s ceiling
IQ4_XS: 8.56B x 4.43 / 8 = 4.74 GB per token -> ~16 tok/s ceilingThe PC rows assume every routed expert sits in RAM. A bigger GPU holds some experts and raises the ceiling. Four DIMMs often run slower than two, so check your board’s rated speed.
Software Support, Checked 22 October 2026
| Runtime | Status | Source |
|---|---|---|
| llama.cpp | Supported. Architecture bailingmoe3 added in PR #26608, merged 17 August 2026 for Ling-3.0-flash | GitHub PR |
| llama.cpp build to use | b11571+ (AtomicChat card), b11627+ (bartowski card) | GGUF cards |
| LM Studio, koboldcpp, Jan | Listed by bartowski’s card. We did not test them | bartowski card |
| MLX (mlx-lm) | Code maps bailing_hybrid + BailingMoeV3ForCausalLM to bailing_moe_v3.py. No Ling-3.1-flash MLX quant on Hugging Face | mlx-lm source, HF search |
| Ollama | No ling-3.1-flash library page (404) | ollama.com |
| SGLang, vLLM | inclusionAI’s recommended path, for 8-GPU servers | model card |
Run bartowski’s files with:
llama-server -hf bartowski/Ling-3.1-flash-GGUF:Q2_K --spec-type draft-mtp -c 262144On a PC, add --n-cpu-moe and lower it until the GPU is full. Our —n-cpu-moe guide shows how.
What to Buy
Prices as of October 2026, from our price reference and apple.com.
| Option | Price | What it runs |
|---|---|---|
| Mac Studio M5 Ultra 256GB (64-core GPU) | $9,499 (apple.com, 22 October) | Q2_K, full context |
| RTX PRO 6000 96GB | $14,999.99 in stock (Central Computer, 30 September) | IQ4_XS, with 256GB of RAM |
| RTX 5090 32GB | marketplace from $6,911 (17 October) | IQ3_XXS, with 256GB of RAM |
| RTX 3090 24GB (used) | $1,518 average (13 October) | IQ2_XXS, with 192GB of RAM |
| DDR5 128GB kit (2 x 64GB) | from $2,099.99 (Newegg, 1 October) | half of a 256GB build |
The Mac Studio is the simplest buy. One box holds Q2_K and the full 262K context. No offload tuning.
For better quants, a GPU workstation wins. The RTX PRO 6000 96GB with 256GB of DDR5 holds IQ4_XS. That is two quant steps above what the 256GB Mac holds. It needs a board that takes 4 x 64GB DIMMs.
On a smaller budget, keep the trunk on the GPU. An RTX 5090 32GB or a used RTX 3090 24GB holds the 16.4B-weight trunk at 2-4 bits. The RAM holds the experts and sets the speed.
Do not buy a 128GB machine for this model. A DGX Spark, a Strix Halo box or a 128GB Mac cannot load any llama.cpp file of it.
Honest Caveats
- We did not run the model. Every fit row is arithmetic from file sizes, config.json and the llama.cpp source.
- The only speed number is from a custom runtime. No llama.cpp tok/s for Ling-3.1-flash was published where we looked.
- The 2 GiB compute buffer is a budget. Check the buffer sizes in your
llama-serverlog. - The 256GB Mac limit is AtomicChat’s suggestion. We did not find a log of the default limit on a 256GB M5 Ultra.
- “Trunk” counts use the card’s 25B figure. AtomicChat’s card says “about 24B” active. The split shifts by about 1B.
- 2-bit files lose quality. AtomicChat measured 92.72% top-1 agreement for AD-IQ2_M. Test it on your own tasks.
Sources
- Ling-3.1-flash model card: parameters, layers, experts, 1M-context claim, SGLang and vLLM recipes
- config.json: max_position_embeddings, rope_scaling, kv_lora_rank, qk_rope_head_dim, heads, head_dim, short_conv_kernel_size, experts, moe_intermediate_size, first_k_dense_replace
- Hugging Face API: creation date, checkpoint parameter count, downloads
- File trees for bartowski and AtomicChat: byte sizes
- bartowski GGUF card: build b11627, bits per weight, MTP flag, compatible apps
- AtomicChat GGUF card: build b11571, KL divergence and top-1 agreement, memory table, Mac sysctl value
- Baekpica VQ-GGUF card: 107.81 GiB VQ size, DGX Spark speed, measured KDA state, DGX Spark MemTotal
- llama.cpp PR #26608: BailingMoE3 support, merged 17 August 2026
- llama.cpp source, master branch:
src/models/bailingmoe3.cpp(MLA latent cache, KDA recurrent state) - llama.cpp RPC README: “proof-of-concept” status
- llama.cpp issue #21680: 115,448.73 MB Metal working set on a 128GB M5 Max
- mlx-lm source: architecture remap to bailing_moe_v3
- Apple Mac Studio store and specs: $9,499 for 256GB, 512GB “coming late October”, 1.2TB/s
- NVIDIA DGX Spark: 273 GB/s, ConnectX-7 clustering of up to four systems
- OpenClaw DC hardware price reference, last updated 17 October 2026: RTX PRO 6000, RTX 5090, RTX 3090 and DDR5 128GB kit rows
See Also
- Best local LLM for the Mac Studio M5 Ultra: what else fits 256GB
- Best models for the biggest Mac Studio: the 256GB model list
- Which Mac Studio to buy: M5 Max or M5 Ultra
- —n-cpu-moe explained: how to put experts in system RAM
- Can you run Kimi K3 locally?: another model past the 128GB line
- Best models for the DGX Spark: 1 to 4-Spark setups
- Is the DGX Spark worth it?: the 128GB GB10 box, priced
- Used EPYC servers for MoE inference: the many-channel RAM route
Need OpenClaw fixed live?
Remote rescue sessions for gateway, auth, tunnel, VPS, and model access problems.
See Rescue Session