Can I Run Mistral Large 4 Locally? (2026): Only at 256GB+
Mistral made Mistral-Large-4-1T-A52B ("Le Chonk") public on Hugging Face on 29 October 2026. It is a 1-trillion-parameter mixture-of-experts model with 52B active parameters, a 1M-token context and image input. llama.cpp support merged the same day. We summed every GGUF file in bytes from the Hugging Face API and did the cache arithmetic from config.json. The short answer: the smallest file is 217 GiB. Nothing under 256GB runs it, and 256GB runs only the 1-bit file.
Bottom Line
- Only with 256GB or more. The smallest GGUF, Unsloth’s UD-IQ1_S, is 217.01 GiB. No 128GB machine loads any file.
- 256GB means 1 bit. A 256GB Mac Studio M5 Ultra holds UD-IQ1_S with a 131K context. Unsloth measured it at 90.6% top-1 agreement with the original.
- 2 bits need about 300GB, 3 bits about 400GB. Mistral’s own NVFP4 release is 568.15 GiB.
- The cache is small. The model compresses its cache (MLA). Our estimate is 81 KiB per token: 2.5 GiB at 32K.
- A GPU plus RAM needs a 32GB card. About 20B always-read weights take ~21.5 GB at 8 bits. A 24GB card has no room left for the cache.
- Support is one day old. llama.cpp merged it on 29 October. Ollama is cloud-only. mlx-lm support is an open pull request.
- If you own 256GB, try it, but do not buy for it. A smaller model at 3-4 bits on the same box is the safer daily driver.
Ready to buy? See the tested hardware list with current prices.
What Mistral Large 4 Is
Figures come from the model card, its config.json and the Hugging Face API, read on 29 October 2026.
| Spec | Value |
|---|---|
| Repo | mistralai/Mistral-Large-4-1T-A52B |
| Released | 29 October 2026 (Le Trim card: “release day”); the repo was created on 30 September |
| Total parameters | ~1T; our count from the BF16 GGUF is 1,049.4B |
| Active per token | 49B, or 52B with embeddings and output layer (card) |
| Layers | 72: 1 dense + 71 MoE |
| Experts | 128 routed per MoE layer, 4 active, plus 1 shared |
| Attention | MLA (compressed cache, rank 512) with a sparse-attention indexer |
| Context | 1,048,576 tokens |
| Inputs | Text and images; text output |
| Reasoning | low, medium, high (default); it cannot be turned off |
| Licence | Mistral Ecosystem Open-Weight License (modified MIT; a hosted model service needs a separate licence) |
| Official checkpoints | FP8 (987.72 GiB), NVFP4 (568.15 GiB) |
Our count matches the card. The 4 active routed experts are 32.16B per token: 4 x 3 x 6,144 x 6,144 x 71 layers. The rest of the model is about 20.2B: attention, the shared expert, the dense first layer, the routers, the token table and the output head. 32.16B + 20.2B is about 52B.
What the card claims. Mistral reports 48% on the Artificial Analysis Coding Agent Index, “effectively tied” with GLM-5.3. It reports 26 on the Intelligence Index v5, the highest for a Western open model. The card says Chinese open models score higher on that index. We ran no benchmark.
Every GGUF Size, in Bytes
We summed every file in the Hugging Face file trees on 29 October 2026. GiB = bytes / 1,073,741,824. Split files are added together.
| File | Repo | Bytes | GiB | Bits per weight |
|---|---|---|---|---|
| UD-IQ1_S | unsloth | 233,009,564,322 | 217.01 | 1.78 |
| UD-IQ1_M | unsloth | 259,584,674,562 | 241.76 | 1.98 |
| IQ2_XXS | ggml-org | 297,676,048,832 | 277.23 | 2.27 |
| UD-IQ2_XXS | unsloth | 302,467,238,722 | 281.69 | 2.31 |
| MLX 2-bit (text only) | mlx-community | 338,131,927,242 | 314.91 | — |
| UD-Q2_K_XL | unsloth | 378,254,511,138 | 352.28 | 2.88 |
| Q2_K | ggml-org | 380,706,433,472 | 354.56 | 2.90 |
| UD-IQ3_XXS | unsloth | 423,515,245,634 | 394.43 | 3.23 |
| MLX 3-bit (text only) | mlx-community | 459,251,397,010 | 427.71 | — |
| UD-Q3_K_XL | unsloth | 492,746,427,746 | 458.91 | 3.76 |
| UD-Q4_K_XL | unsloth | 604,787,428,898 | 563.25 | 4.61 |
| NVFP4 GGUF (original experts) | ggml-org | 630,480,508,160 | 587.18 | — |
| Q4_K_M | ggml-org | 634,802,244,608 | 591.21 | 4.84 |
| Q8_0 | unsloth / ggml-org | 1,115,150,340,850 | 1,038.56 | 8.50 |
| BF16 | unsloth / ggml-org | 2,098,861,022,194 | 1,954.72 | 16.00 |
Bits per weight = bytes x 8 / 1,049.4B. The vision projector (mmproj) is a separate 3.17-3.21 GiB file. The ggml-org IQ2_XXS keeps attention, the indexer, the shared experts, the dense layer and the embeddings at Q8_0. Only the routed experts drop to 2 bits.
How much quality each step costs. Unsloth measured each of its files against the original, across 7 task groups. “Top-1” is how often the quant picks the same next token:
| File | GiB | Mean KL divergence | Top-1 agreement |
|---|---|---|---|
| UD-IQ1_S | 217.01 | 0.1263 | 90.6% |
| UD-IQ1_M | 241.76 | 0.0919 | 92.3% |
| UD-IQ2_XXS | 281.69 | 0.0566 | 93.9% |
| UD-Q2_K_XL | 352.28 | 0.0298 | 95.6% |
| UD-IQ3_XXS | 394.43 | 0.0217 | 96.2% |
| UD-Q3_K_XL | 458.91 | 0.0162 | 96.6% |
| UD-Q4_K_XL | 563.25 | 0.0071 | 97.8% |
The 1-bit file drifts four times as much as the 2-bit K file. KL divergence is 0.1263 at UD-IQ1_S and 0.0298 at UD-Q2_K_XL. Treat 1 bit as a way to try the model, not as its real quality.
Why the Cache Costs So Little
Mistral Large 4 uses multi-head latent attention (MLA). Each layer stores one compressed vector per token, not full keys and values.
config.json: kv_lora_rank 512, qk_rope_head_dim 64, 72 layers
f16: (512 + 64) x 2 bytes x 72 layers = 82,944 bytes per token (about 81 KiB)
32,768 tokens = 2.5 GiB
131,072 tokens = 10.1 GiB
1,048,576 tokens = 81.0 GiBThis is our estimate from config.json. The sparse-attention indexer also keeps a small key per token in some layers. Read the real cache size in your llama-server log before you push the context up.
The full budget. Weights + cache + a compute buffer. We budget 2 GiB for the buffer. We did not measure it.
Fit Table by Hardware
PC rows use VRAM plus system RAM, minus about 8 GiB for the OS. Every row is our arithmetic unless it says community-reported.
| Hardware | Usable memory | Best file | Context | Notes |
|---|---|---|---|---|
| Any 128GB machine | ~107-117 GiB | none | — | Smallest file is 217.01 GiB |
| 24GB GPU + 256GB DDR5 | ~272 GiB | UD-IQ1_M (241.76) | short | Always-read part fills the card |
| 2 x DGX Spark | ~234 GiB | UD-IQ1_S (217.01) | 32K | llama.cpp RPC; no run found |
| 256GB Mac Studio M5 Ultra | 240 GiB with raised limit | UD-IQ1_S (217.01) | 131K | 1 bit only |
| 32GB GPU + 256GB DDR5 | ~280 GiB | UD-IQ1_M (241.76) | 131K | Experts in RAM |
| 4 x RTX PRO 6000 | 384GB VRAM | Le Trim 50% NVFP4 (298.57) | long | Pruned build; vLLM; no run found |
| 512GB Mac Studio | not orderable | MLX 3-bit (427.71) | — | Apple does not sell it yet |
| 8 x RTX PRO 6000 | 768GB VRAM | Official NVFP4 (568.15) | long | 53.3 tok/s (community) |
A 256GB Mac Studio holds the 1-bit file and a long context. UD-IQ1_S (217.01 GiB) + 10.1 GiB cache + 2 GiB buffer is about 229 GiB. That is under 240 GiB. Raise the limit first with sudo sysctl iogpu.wired_limit_mb=245760. That leaves macOS 16 GiB. UD-IQ1_M (241.76 GiB) does not fit at any context.
Unsloth’s “fits 256GB” line needs the raised limit. Their guide says UD-IQ1_S “should fit on 256GB RAM including a Mac Studio or 2x NVIDIA DGX Sparks”. On a Mac it fits only after the sysctl change. On two Sparks it fits with a short context only: 217 GiB of weights in about 234 GiB of usable memory.
Two Sparks are untested. The 117.2 GiB per Spark is the usable figure from a Beam community card. llama.cpp RPC splits the layers over the network link. We found no published Mistral Large 4 run on two Sparks.
The 24GB GPU route is tight. The 20.2B non-routed weights take about 21.5 GB at 8 bits. Unsloth may store some of them smaller, but plan for a 32GB card. Our —n-cpu-moe guide shows how to put the experts in system RAM.
The 512GB Mac Studio is the box this model wants. It would hold the 3-bit MLX file (427.71 GiB). The mlx-lm pull request says the same: “The 3-bit build (459 GB) fits on one 512GB Mac.” On 29 October, apple.com still showed no 512GB configuration to order.
Is the Pruned Le Trim Build Worth It?
Le Trim removes routed experts from all 71 MoE layers. The router still picks 4 experts, so active parameters stay near 52B. It is smaller, not faster.
| Build | Experts per layer | Size | Perplexity (uploader) |
|---|---|---|---|
| Official NVFP4 | 128 | 568.15 GiB | 1.1076 |
| Le Trim 25% | 96 | 478 GB (card) | 1.1107 (+0.28%) |
| Le Trim 50% | 64 | 298.57 GiB | 1.1234 (+1.4%) |
One fixed text is a thin test. The uploader measured perplexity on one held-out text, in vLLM. The Beam REAP prune also looked fine on perplexity and lost 14 points on MMLU-Pro. Test Le Trim on your own tasks before you buy cards for it. It runs in vLLM and SGLang, not in llama.cpp.
How Fast Will It Be?
We found only one speed figure. The model went public on the day we checked.
| Setup | Build | Decode |
|---|---|---|
| 8 x RTX PRO 6000, vLLM, batch 1 | Official NVFP4 | 53.3 tok/s (community) |
| 8 x RTX PRO 6000, vLLM, batch 1 | Le Trim 25% NVFP4 | 51.9 tok/s (community) |
Both rows come from the Le Trim card. We found no Mac, Spark or GPU-plus-RAM speed. The rows below are bandwidth ceilings, not predictions. Real speed lands below them.
Routed experts read per token (our count from config.json):
4 experts x 3 x 6,144 x 6,144 x 71 layers = 32.16B (in RAM if offloaded)
PC, experts in RAM, dual-channel DDR5-4800 = 76.8 GB/s:
UD-IQ1_S, 1.78 bits/weight: 32.16B x 1.78 / 8 = 7.14 GB -> ~11 tok/s ceiling
UD-IQ1_M, 1.98 bits/weight: 32.16B x 1.98 / 8 = 7.96 GB -> ~10 tok/s ceilingThe bits per weight are file averages. Four DIMMs often run slower than two, which lowers the ceiling.
Expect a Mac to run it slower than Beam. Beam 1.0 reads 23.4B active weights per token and measured about 29 tok/s on a 256GB M5 Ultra. Mistral Large 4 reads 52B. More than twice the reads per token means fewer tokens per second on the same bandwidth.
Software Support, Checked 29 October 2026
| Runtime | Status | Source |
|---|---|---|
| llama.cpp | PR #31376 merged 29 October 2026. b11998 is the first release tag that contains it | GitHub |
| Ollama | Only mistral-large-4:cloud. No local tag | ollama.com |
| MLX | mlx-lm PR #2018 (text model) open. mlx-community 2/3/4-bit files already uploaded | GitHub, Hugging Face |
| vLLM | Recommended by Mistral. Docker image vllm/vllm-openai:mistral-large-4 | Model card |
| SGLang | Supported, with a cookbook | Model card |
| DSpark draft | Ships inside the checkpoint. Mistral measured “up to 1.9x” throughput in vLLM/SGLang | Model card |
Run the 1-bit file on a 256GB Mac with:
sudo sysctl iogpu.wired_limit_mb=245760
llama-server -hf unsloth/Mistral-Large-4-1T-A52B-GGUF:UD-IQ1_S --ctx-size 32768 \
--temp 1.0 --top-p 0.95 --top-k 0 --min-p 0.0 \
--chat-template-kwargs '{"reasoning_effort":"medium"}'Use release b11998 or newer. Older builds do not know the mistral_large4 architecture. Mistral recommends temperature 1.0 and top-p 0.95 for every reasoning level. Use low for quick chat, because reasoning cannot be turned off.
What to Buy
Prices as of October 2026, from our price reference. The read date is in brackets.
| Option | Price | What it runs |
|---|---|---|
| Mac Studio M5 Ultra 256GB (64-core GPU, 1TB) | $9,499 (apple.com, 22 October) | UD-IQ1_S with 131K context |
| 2 x NVIDIA DGX Spark 128GB | check current listings | UD-IQ1_S at short context; untested |
| 4 x RTX PRO 6000 96GB | $14,999.99 per card (Central Computer, 30 September) | Le Trim 50% NVFP4 in vLLM; untested |
| Mac Studio 512GB | not orderable on 29 October | MLX 3-bit |
Do not buy hardware for this model yet. Every affordable route is 1 bit, and the 1-bit file drifts four times as much as 2-bit. The 512GB Mac Studio that would run 3 bits cannot be ordered yet.
Already own a 256GB Mac Studio? Try UD-IQ1_S. It costs you a 233 GB download. For daily use on the same box, Beam 1.0 at 3 bits keeps 96.13% top-1 agreement and leaves room for a 1M context.
Two Sparks are the CUDA experiment. Two NVIDIA DGX Spark 128GB boxes give about 234 GiB, enough for UD-IQ1_S at a short context over llama.cpp RPC. Nobody has published that run yet. Buy them for 120B-class models such as Ornith 1.5 122B, and treat Mistral Large 4 as a test.
The fast route is a rack of cards. Four RTX PRO 6000 96GB cards hold the 298.57 GiB Le Trim 50% build in VRAM. That is about $60,000 of GPUs for a pruned model with one perplexity test. Eight cards run the official NVFP4 at 53.3 tok/s (community-reported).
Honest Caveats
- We did not run the model. Every fit row is arithmetic from file sizes and config.json.
- The model is one day old. The only speed numbers come from one uploader’s card.
- The cache figure is our estimate. It ignores the sparse-attention indexer keys. Check your
llama-serverlog. - The 2 GiB compute buffer is a budget. We did not measure it.
- The DGX Spark memory figure is second-hand. The 117.2 GiB comes from a community card for another model.
- The KL table is Unsloth’s. Their text and method are not the same as other uploaders’ tables.
- The benchmark claims are Mistral’s. We have not seen independent results yet.
Sources
- Mistral-Large-4-1T-A52B model card: parameters, experts, context, licence, reasoning levels, benchmark claims, DSpark, recommended servers and settings
- config.json: hidden_size, kv_lora_rank, qk_rope_head_dim, num_hidden_layers, first_k_dense_replace, n_routed_experts, num_experts_per_tok, moe_intermediate_size
- File trees for unsloth GGUF, ggml-org GGUF, ggml-org NVFP4 GGUF, mlx-community 2-bit and 3-bit, and the official FP8 and NVFP4 repos: byte sizes
- Unsloth Mistral Large 4 guide: memory table, KL divergence and top-1 agreement, run commands
- ggml-org GGUF card: per-tensor quant types of IQ2_XXS, imatrix, reasoning cannot be turned off
- Le Trim card: pruning, perplexity, 8 x RTX PRO 6000 speeds (community-reported)
- llama.cpp PR #31376 and release b11998: support and merge date
- mlx-lm PR #2018: MLX support status, 512GB note
- Ollama library: mistral-large-4: cloud tag only
- Our Beam 1.0 page: M5 Ultra wired limit, the 117.2 GiB Spark figure, Beam speeds
- apple.com Mac Studio store page, read 29 October 2026: no 512GB configuration offered
- OpenClaw DC hardware price reference, last updated 28 October 2026: Mac Studio M5 Ultra and RTX PRO 6000 rows
See Also
- Can I run MiniMax M3.1 Flash locally? — the 428B MoE that runs at 3 bits on 192GB
- Can I run Beam 1.0 501B locally? — the 501B MoE that runs at 3 bits on a 256GB Mac
- Can you run Kimi K3 locally? — the other 1T-class open model
- Best local LLM for a Mac Studio M5 Ultra 256GB — what to run on the 256GB box instead
- Which Mac Studio to buy for local LLMs — every M5 configuration priced
- Best models for the DGX Spark — 1 to 4-Spark setups
- Four RTX PRO 6000 Max-Q cards in one workstation — the 384GB VRAM build
- —n-cpu-moe explained — how to put experts in system RAM