Can I Run Flint Locally? (2026): Yes, on a 24GB Card
Recursive put Flint on Hugging Face on 21 October 2026. It is a 39.8B mixture-of-experts coding model with about 2.5B parameters active per token, Apache 2.0, trained from scratch. We summed every GGUF file in bytes from the Hugging Face API, did the cache arithmetic from config.json, and checked llama.cpp, Ollama, LM Studio, MLX and vLLM. The short answer: a 24GB card runs a 4-bit file fully on the GPU, and a 32GB card holds the full 262K context.
Bottom Line
- Yes. A 24GB card runs Flint at 4 bits, fully on the GPU. bartowski’s IQ4_XS is 20.90 GiB. With a 32K cache the total is about 22.9 GiB.
- A 32GB card holds the full 262K context. The official NVFP4 file (21.29 GiB) plus the 7.0 GiB cache is about 29.4 GiB.
- A 16GB card works with system RAM. Put some experts in RAM with
--n-cpu-moe. One owner measured ~49 tok/s on a 16GB card (community-reported). - The cache is small. Only 14 of 49 layers keep a full KV cache: 28,672 bytes per token, so 262K costs 7.0 GiB.
- Macs need 32GB for 4 bits. A 48GB Mac holds the official INT4 GGUF with the full context. A 64GB M5 Pro Mac mini ran 4-bit at 87 tok/s at 20K context (community-reported).
- Every common runtime loads it today: llama.cpp (b11803+), Ollama (v0.40.9+ on Linux and Windows), LM Studio, MLX from source, and vLLM from source.
Ready to buy? See the tested hardware list with current prices.
What Flint Is
Figures come from the model card, its config.json and the Hugging Face API, read on 26 October 2026.
| Spec | Value |
|---|---|
| Repo | RecursiveSI/flint |
| Maker | Recursive |
| Created on Hugging Face | 21 October 2026 |
| Architecture | FlintForCausalLM, a new architecture. Trained from scratch (card) |
| Type | Mixture of experts (MoE), not dense |
| Parameters | 39,785,287,456 (BF16, HF API) |
| Active per token | ~2.5B including the output head; ~2.2B without it (card) |
| Layers | 49: 2 dense + 47 MoE |
| Experts | 256 routed + 1 shared per MoE layer, 6 routed active |
| Attention | 14 full-attention + 35 sliding-window layers (window 513 tokens) |
| KV heads | 4 KV heads, 32 query heads, head_dim 128 |
| Context | 262,144 tokens, prompt and output together |
| Inputs | Text only |
| Licence | Apache 2.0 |
It is a sparse model. Our count from config.json puts the routed experts at 37.85B of the 39.8B parameters. Only 0.89B of those are read per token. The other weights (attention, shared expert, dense layers, router, embedding and output head) total about 1.94B.
What the card claims. These are Recursive’s own numbers, not ours:
| Benchmark (card) | Flint BF16 | Flint NVFP4 |
|---|---|---|
| Terminal-Bench 2.1 | 54.9 | 53.7 |
| SWE-bench Pro v2 | 89.7 | 88.5 |
| SWE-bench Verified | 74.2 | 74.1 |
Treat them with care. Community testers on Hugging Face call it fast but uneven on long agent tasks. One Q8_0 user reports it deleted its own report file during a cleanup step. We did not run any benchmark.
Every GGUF Size, in Bytes
We read every file in the Hugging Face file trees on 26 October 2026. GiB = bytes / 1,073,741,824. Split files are added together. Flint is text-only, so there is no vision projector.
| File | Repo | Bytes | GiB |
|---|---|---|---|
| UD-IQ1_M | unsloth | 11,352,241,600 | 10.57 |
| UD-IQ2_XXS | unsloth | 11,855,541,696 | 11.04 |
| UD-Q2_K_XL | unsloth | 13,562,207,680 | 12.63 |
| UD-IQ3_XXS | unsloth | 14,535,427,520 | 13.54 |
| UD-Q3_K_XL | unsloth | 18,494,977,472 | 17.22 |
| Q3_K_M | bartowski | 19,529,151,712 | 18.19 |
| IQ4_XS | bartowski | 22,446,290,144 | 20.90 |
| NVFP4 | RecursiveSI (official) | 22,863,084,160 | 21.29 |
| Q4_K_M | ggml-org | 23,387,911,136 | 21.78 |
| Q4_K_M | lmstudio-community | 24,166,841,184 | 22.51 |
| UD-Q4_K_XL | unsloth | 25,054,688,704 | 23.33 |
| Q4_K_M | bartowski | 25,346,651,360 | 23.61 |
| INT4 (Q4_1) | RecursiveSI (official) | 25,992,966,976 | 24.21 |
| UD-Q5_K_XL | unsloth | 29,919,491,520 | 27.86 |
| Q5_K_M | bartowski | 30,809,732,320 | 28.69 |
| Q6_K | bartowski | 35,199,071,456 | 32.78 |
| UD-Q6_K_XL | unsloth | 36,319,999,424 | 33.83 |
| Q8_0 | ggml-org / bartowski | 42,350,359,520 | 39.44 |
| BF16 | ggml-org | 79,625,761,760 | 74.16 |
Non-GGUF builds: the official MLX INT4 repo is 21.02 GiB and mlx-community’s mixed 3/4-bit build is 16.61 GiB.
Three files called Q4_K_M differ by 1.83 GiB. ggml-org’s card says its Q4_K_M keeps the small non-expert tensors at Q8_0. bartowski’s card says his files use a layout computed for this model. Compare files by bytes, not by name.
The official files are trained, not just rounded. Recursive’s launch post says the quantized model was distilled from the 16-bit model. It says the final NVFP4 model scores “within 0.7 points” of 16-bit. The INT4 GGUF stores the trained INT4 codes from the MLX export.
Even the output head is 4-bit. The launch post says Recursive quantized the output layer to 4 bits, “against common practice”. Our count makes that layer 0.31B of the 2.5B read per token, about 12%. In a 2.5B-active model the output head is a large part of every token’s reads.
Why the Context Costs So Little
Only the 14 full-attention layers keep a cache that grows with context. The 35 sliding-window layers keep only the last 513 tokens.
1. The full-attention cache. Config.json gives num_key_value_heads 4 and head_dim 128. layer_types lists 14 full_attention layers.
f16: 2 (K and V) x 4 heads x 128 x 2 bytes x 14 layers = 28,672 bytes per token
32,768 tokens = 0.875 GiB
65,536 tokens = 1.75 GiB
131,072 tokens = 3.5 GiB
262,144 tokens = 7.0 GiB
q8_0 cache: about half of each figure2. The sliding-window cache. llama.cpp sizes it at the window plus one micro-batch, padded to 256 cells (llama-kv-cache-iswa.cpp). With the default 512 micro-batch that is 1,280 cells.
35 layers x 2,048 bytes x 1,280 cells = 87.5 MiB, at any context length3. Do not set --swa-full. It gives the 35 window layers a full-length cache too. At 262K that adds about 17.5 GiB. Recursive’s GGUF cards say to leave it unset.
4. The full budget. Weights + cache + 0.09 GiB window cache + a compute buffer. We budget 1 GiB for the buffer. We did not measure it.
Fit Table by Hardware
GPU rows assume the whole card is usable, minus about 0.5 GiB for the driver. PC rows with RAM use VRAM plus system RAM. Every row is our arithmetic unless it says community-reported.
| Hardware | Best file | Context | Total | Notes |
|---|---|---|---|---|
| 16GB GPU, all on GPU | UD-Q2_K_XL (12.63) | 32K | ~14.6 GiB | 2-bit; quality cost |
| 16GB GPU + 32GB DDR5 | UD-Q4_K_XL (23.33) | 16K+ | split | Experts in RAM; ~49 tok/s (community) |
| 24GB GPU, all on GPU | IQ4_XS (20.90) | 32K f16 / 64K q8_0 | ~22.9 GiB | The verdict |
| 24GB GPU, long context | UD-Q3_K_XL (17.22) | 128K f16 / 262K q8_0 | ~21.8 GiB | 3-bit |
| 24GB GPU + 32GB DDR5 | Q4_K_M (21.78) | 262K | split | Experts in RAM |
| 32GB GPU (RTX 5090) | NVFP4 (21.29) or Q4_K_M (21.78) | 262K | ~29.4-29.9 GiB | Full context on the GPU |
| 32GB GPU, more bits | Q5_K_M (28.69) | 32K | ~30.6 GiB | Tight |
| 2 x 32GB GPUs | Q8_0 (39.44) | 262K | ~47.5 GiB | Ran on 2 x R9700 (community) |
| 32GB Mac | MLX INT4 (21.02) or UD-Q3_K_XL (17.22) | 32K / 64K | ~22.9 / ~20.1 GiB | Default limit is an estimate |
| 48GB Mac | Official INT4 GGUF (24.21) | 262K | ~32.3 GiB | Recursive’s own minimum |
| 64GB Mac | Q6_K (32.78) | 262K | ~40.9 GiB | 4-bit ran at 87 tok/s (community) |
The 24GB card is the clean tier. IQ4_XS + 0.875 GiB cache + 0.09 GiB window cache + 1 GiB buffer is 22.87 GiB. A q8_0 cache doubles that to 64K for the same memory. A used RTX 3090 24GB runs it with no offload.
The 32GB card runs the official file at full context. NVFP4 + 7.0 GiB cache + buffers is about 29.4 GiB. One owner ran the NVFP4 GGUF on a 48GB RTX PRO 5000 at full context. They saw about 26 GB of VRAM in use and ~115 tok/s decode (community-reported). That is below our arithmetic, so our totals are conservative.
Recursive’s “48 GB” line is a recommendation, not a floor. The NVFP4 card recommends “at least 48 GB of GPU memory”. The file and the full cache need about 29 GiB. Recursive tested it on B300, RTX PRO 6000 Blackwell, RTX Spark and DGX Spark, not on a 32GB card.
Mac limits. A 48GB M4 Pro reports a 40,200.90 MB Metal working set in llama.cpp issue #20141. llama.cpp prints that in units of 10^6 bytes, so it is 37.44 GiB. That is 78% of 48 GiB. At the same 78%, a 32GB Mac gets about 25 GiB (our estimate). We found no logged figure for a 64GB Mac. You can raise the limit with sudo sysctl iogpu.wired_limit_mb=N, with N in MiB.
The 16GB route works because the experts are small. The always-read weights are about 1.94B, near 2 GiB at Q8_0. A 16GB card holds them, the cache and about half the experts. The rest live in RAM.
How Fast Will It Be?
We found four owners who posted numbers on local hardware. All are community-reported, from the Hugging Face discussion pages linked in Sources.
| Setup | Prompt processing | Generation |
|---|---|---|
RX 7800 XT 16GB + 32GB DDR5-6200, UD-Q4_K_XL, -ncmoe 24, Vulkan | 1,886 tok/s (pp2048) | 49.0 tok/s |
| Same, at 16K depth | 1,288 tok/s | 48.3 tok/s |
| RTX PRO 5000 48GB, official NVFP4, full context | not given | ~115 tok/s |
| Mac mini M5 Pro 64GB, Ollama 4-bit, short prompt | not given | 115 tok/s |
| Same, 20K-token codebase | 20K read in 11 s | 87 tok/s |
RTX 3090, UD-IQ2_XXS, ncmoe 20, q4_0 cache | not given | 50-60 tok/s short, 10-15 tok/s after 30 min |
Unsloth’s guide claims “125+ tokens/sec on an M4 Pro MacBook Pro”. That is a vendor claim. We found no matching log.
Offloaded experts cost little here. Each token reads only 0.89B routed-expert weights.
Routed weights read per token (our count from config.json):
6 experts x 3 x 2,048 x 512 x 47 layers = 0.89B
At ~4.5 bits per weight (Q4_K):
0.89B x 4.5 / 8 = 0.50 GB per token
Dual-channel DDR5-4800 = 76.8 GB/s -> ~154 tok/s ceiling if every expert lived in RAMThat is a ceiling, not a prediction. The 16GB owner’s 49 tok/s with half the experts in RAM fits that picture.
Speed drops on long agent runs. The RTX 3090 owner saw 50-60 tok/s fall to 10-15 tok/s after 30 minutes of long prompts. That run used a 2-bit file and an 8GB VRAM budget. More of the model on the GPU should hold speed better.
The current llama.cpp code is not the final speed. The llama.cpp PR says its expert-activation scaling costs “10-30%”. It says follow-up PRs will address it.
Software Support, Checked 26 October 2026
| Runtime | Status | Source |
|---|---|---|
| llama.cpp | Supported. PR #30928 merged 21 October 2026; first release b11803 (21 October) | GitHub |
| llama.cpp chat template | Needs PR #30929 (hex string escapes), merged 21 October, inside b11803. Use --jinja | GitHub, unsloth thread |
| llama.cpp multi-GPU | --split-mode tensor is disabled for Flint (PR #31081, 24 October). Layer split still works | GitHub, 2 x R9700 report |
| Ollama | flint (23GB default), plus nvfp4, q4_1, qat-int4 and mlx-bf16 (80GB) tags. macOS from v0.40.8; Linux and Windows from v0.40.9 | ollama.com, GitHub releases |
| LM Studio | Lists recursivesi/flint, “Minimum system memory 24 GB”, built on lmstudio-community GGUFs | lmstudio.ai |
| MLX | mlx-lm PR #1998 merged 24 October. The latest release, v0.32.2, is from 16 October, so install from source. Official flint-int4 (21.02 GiB) runs in Recursive’s rec engine | GitHub, HF |
| vLLM | PR #62938 merged 22 October. The latest release, v0.32.0, is from 18 October, so it is not in a release yet | GitHub |
| SGLang | Recursive’s official kit, one B300 or H200 per replica | model card |
| Transformers | ”Transformers modeling code is not included yet” | model card |
Run the official NVFP4 file on a 32GB Blackwell card with Recursive’s own command:
llama-server -m flint-nvfp4.gguf --alias flint -ngl all -c 262144 -np 1 \
-fa on -ctk f16 -ctv f16 --jinja --temp 1 --top-p 0.95 --top-k 20 --min-p 0On a 24GB card, use IQ4_XS with -c 32768. On a 16GB card, add -ngl 99 --n-cpu-moe 47 and lower --n-cpu-moe until the GPU is full. Our —n-cpu-moe guide shows how.
Tool calls need care outside rec. Recursive’s rec engine constrains tool calls with a grammar. One integrator saw 41 of 85 tool calls fail in mlx-vlm without one, and 0 of 50 with one (community-reported). llama.cpp issue #31000 reports tool-call XML left inside reasoning. It is still open.
What to Buy
Prices as of October 2026, from our price reference. The read date is in brackets.
| Option | Price | What it runs |
|---|---|---|
| RTX 5060 Ti 16GB | $789 at B&H (24 September); $589-805 range (August) | UD-Q4_K_XL with experts in RAM |
| RTX 3090 24GB (used) | $1,541 average, 317 listings (resaleprices.com, 22 October) | IQ4_XS fully on the GPU, 32K |
| RTX 5090 32GB | from $6,987, 92 marketplace listings (GPU Poet, 23 October) | NVFP4, full 262K context |
| DDR5 64GB (2 x 32GB) | $919.99 (Newegg, 1 October) | Room for experts on a 16GB card |
| Mac mini M5 Pro 64GB | $2,699 (apple.com, 1 October) | Q6_K with full context |
For most people, buy 24GB. A used RTX 3090 runs a 4-bit Flint with no offload and no tuning. It costs under a quarter of a marketplace 5090.
Already own a 16GB card? Do not replace it for Flint. Add RAM and offload experts. If you are buying new at 16GB, the RTX 5060 Ti 16GB runs the 4-bit file with experts in RAM. Pair it with 32GB or more of system RAM.
Buy 32GB only for the full context on the GPU. The RTX 5090 32GB holds the official NVFP4 file and all 262K tokens. Marketplace prices are far above its $1,999 list, so check the price first.
On a Mac, buy 48GB or more. Recursive’s INT4 GGUF card says “at least 48 GB of unified memory”. The Mac mini M5 Pro comes in 48GB at $2,299 (apple.com, 22 September) and 64GB at $2,699 (apple.com, 1 October). Recursive says its rec engine is optimized for M4 Pro and M5 Pro/Max chips.
Honest Caveats
- We did not run the model. Every fit row is arithmetic from file sizes and config.json.
- Every speed number is one owner’s post. No two runs used the same file and settings.
- The 1 GiB compute buffer is a budget. Check the buffer sizes in your
llama-serverlog. - The 32GB Mac limit is an estimate. It scales a 48GB Mac’s logged figure by the same ratio.
- We did not confirm that NVFP4 runs fast on pre-Blackwell cards. On a 24GB RTX 3090, use a standard GGUF.
- The benchmark table is Recursive’s. Community tests are mixed. Test the model on your own tasks.
Sources
- Recursive launch post: NVFP4 distillation, 4-bit output head, 0.7-point claim, rec engine targets
- Flint model card: architecture, active parameters, layer split, benchmark claims, SGLang kit, Transformers note
- config.json: layer_types, sliding_window, num_key_value_heads, head_dim, experts, moe_intermediate_size, shared_expert_intermediate_size, max_position_embeddings
- Hugging Face API: creation date, parameter count
- File trees for unsloth, bartowski, ggml-org, lmstudio-community, official NVFP4, official INT4 GGUF, official MLX INT4 and mlx-community: byte sizes
- Official NVFP4 GGUF card and INT4 GGUF card: 48 GB recommendations, tested hardware, llama-server commands,
--swa-fullnote - ggml-org GGUF card and bartowski GGUF card: Q8_0 non-expert tensors, per-model layouts, compatible apps
- Unsloth Flint guide: M4 Pro speed claim, benchmark table
- Community threads: unsloth discussion #1 (RX 7800 XT, RTX 3090, 2 x R9700), unsloth discussion #5 (NVFP4 distillation quote), Flint discussion #15 (M5 Pro Mac mini), Flint discussion #17 (RTX PRO 5000), Flint discussion #11 (tool-call grammar)
- llama.cpp PR #30928, PR #30929, PR #31081, release b11803, issue #31000, issue #20141 and
src/llama-kv-cache-iswa.cpp - Ollama library: flint and Ollama releases v0.40.8 and v0.40.9
- LM Studio: recursivesi/flint
- mlx-lm PR #1998 and vLLM PR #62938
- OpenClaw DC hardware price reference, last updated 23 October 2026: RTX 5060 Ti, RTX 3090, RTX 5090, DDR5 and Mac mini rows
See Also
- Can I run Sage-A1.5 locally?: the other new agent MoE that fits a 24GB card, with vision and MTP
- Can I run LFM2.5 24B A2B locally?: another 2.5B-active MoE that fits 16GB
- Xing4.0 29B A4B on RTX 3090: the other new coding MoE on a 24GB card
- Best local LLM for the RTX 3090: what else to run on the 24GB tier
- Best local LLM for the RTX 5090: the 32GB tier
- Best local LLM for the RTX 5060 Ti 16GB: the 16GB tier
- —n-cpu-moe explained: how to put experts in system RAM
- Best GPU for a local coding model: why context, not quant, limits local coding