Can I Run Sage-A1.5 Locally? (2026): Yes, on a 24GB GPU
InternScience put Sage-A1.5 on Hugging Face on 15 October 2026. It is a 35B mixture-of-experts agent model built on Qwen3.6-35B-A3B, Apache 2.0, with a vision tower and a built-in MTP head. We summed every GGUF file in bytes from the Hugging Face API, did the cache arithmetic from config.json, and checked llama.cpp, Ollama, LM Studio and MLX. The short answer: a 24GB card runs the 4-bit file fully on the GPU, and a 32GB card holds the full 262K context at 5 bits.
Bottom Line
- Yes. A 24GB card runs Sage-A1.5 at 4 bits, fully on the GPU. Q4_K_M is 20.22 GiB. With a 64K cache the total is about 22.5 GiB.
- A 32GB card holds the full 262K context. Q5_K_M (23.61 GiB) plus the 5.0 GiB cache is about 29.7 GiB.
- A 16GB card works with system RAM. One owner put the experts in 64GB of DDR5 and measured 166 tok/s on an RTX 5080 (community-reported).
- The cache is small. Only 10 of 40 layers keep a KV cache: 20,480 bytes per token, so 262K costs 5.0 GiB.
- Do not take the official Q4_K_M. Imatrix Q4_K_M files are the same size to within 1,056 bytes. One quantizer measured 35-40% lower KL divergence for them (community-measured).
- It is Qwen3.6-35B-A3B underneath. Any runtime that loads Qwen3.6-35B-A3B loads this architecture. llama.cpp supports it with vision and MTP.
Ready to buy? See the tested hardware list with current prices.
What Sage-A1.5 Is
Figures come from the model card, its config.json and the Hugging Face API, read on 26 October 2026.
| Spec | Value |
|---|---|
| Repo | InternScience/Sage-A1.5 |
| Maker | InternScience, Shanghai AI Laboratory |
| Created on Hugging Face | 15 October 2026 |
| Base | Qwen3.6-35B-A3B and Agents-A1 (card) |
| Architecture | Qwen3_5MoeForConditionalGeneration (qwen3_5_moe) |
| Type | Mixture of experts (MoE), not dense |
| Parameters | 35,951,822,704 (BF16, HF API), including vision and MTP |
| Active per token | ~2.95B, our count from config.json |
| Layers | 40: 10 full attention + 30 linear attention (Gated DeltaNet), plus 1 MTP layer |
| Experts | 256 routed + 1 shared per layer, 8 routed active |
| Attention | 16 query heads, 2 KV heads, head_dim 256, output gate |
| Context | 262,144 tokens. The card shows YaRN to 512K in SGLang and vLLM |
| Inputs | Text, image and video (27-layer vision tower) |
| Licence | Apache 2.0 |
What it is for. The card describes a long-horizon agent for research and scientific work. It has two opt-in modes. Agentic Orchestration starts when you register a spawn_subagent tool, and the model sends sub-questions to parallel sub-agents. Verification Mode starts with chat_template_kwargs: {"verification": true}, and it needs a non-empty system message.
It is a sparse model. Our count from config.json puts the routed experts at 32.21B weights. Only 1.01B of those are read per token. The other reads are the linear-attention layers (1.01B), the output head (0.51B) and the full-attention layers (0.27B). The shared expert (0.13B) and the router add the rest. That totals about 2.95B, which matches the “approximately 3B active” on the mlx-community card.
What the card claims. These are InternScience’s own numbers, not ours. The Qwen3.6-35B-A3B column is the base model, from the same table:
| Benchmark (card) | Sage-A1.5 | Qwen3.6-35B-A3B |
|---|---|---|
| GAIA | 95.1 | 78.6 |
| BrowseComp | 81.7 | 67.9 |
| SWE-bench Pro | 61.2 | 49.5 |
| SWE-bench Multilingual | 72.7 | 67.2 |
| IFBench | 81.6 | 64.4 |
| HLE with tools | 55.1 | 36.2 |
The card says the SWE-bench runs used mini-SWE-agent with a 200K-token context. It also says other models’ scores come from their own reports where they exist. We did not run any benchmark.
Every GGUF Size, in Bytes
We read every file in the Hugging Face file trees on 26 October 2026. GiB = bytes / 1,073,741,824. Each file is one part.
| File | Repo | Bytes | GiB |
|---|---|---|---|
| Q2_K | mradermacher | 13,247,184,128 | 12.34 |
| i1-IQ3_M | mradermacher | 15,806,625,024 | 14.72 |
| IQ3_S-exps | na0x2c6 | 16,051,153,760 | 14.95 |
| Q3_K_M | mradermacher | 17,166,660,864 | 15.99 |
| IQ3_S-exps-down4 | na0x2c6 | 17,477,217,120 | 16.28 |
| i1-IQ4_XS | mradermacher | 19,179,523,328 | 17.86 |
| IQ4_XS-exps | na0x2c6 | 19,322,710,880 | 18.00 |
| MXFP4_MOE | prithivMLmods | 20,261,571,136 | 18.87 |
| Q4_K_M | InternScience (official) | 21,713,463,520 | 20.22 |
| Q4_K_M (imatrix) | na0x2c6 | 21,713,464,160 | 20.22 |
| i1-Q4_K_M | mradermacher | 21,713,464,576 | 20.22 |
| Q5_K_M | mradermacher | 25,347,534,080 | 23.61 |
| Q6_K | InternScience (official) | 29,208,731,872 | 27.20 |
| Q8_0 | InternScience (official) | 37,802,150,112 | 35.21 |
| F16 | InternScience (official) | 71,066,995,232 | 66.19 |
| mmproj (vision, f16) | InternScience (official) | 899,282,880 | 0.84 |
| mmproj (vision, Q8_0) | mradermacher | 614,194,144 | 0.57 |
Non-GGUF builds: mlx-community’s OptiQ-4bit language weights are 20.62 GiB, plus a 0.83 GiB vision sidecar and a 0.45 GiB MTP head. InternScience’s official FP8 repo is 36.66 GiB. A community NVFP4 repo (kvallespin) is 22.55 GiB.
The vision tower is a separate file. Every official GGUF repo ships Sage-A1.5-mmproj.gguf (0.84 GiB) next to the model. Leave it out for text-only use. Add it with --mmproj for images.
The GGUFs keep the MTP layer. na0x2c6’s card says all its files keep the built-in MTP (NextN) layer, as the official file does. That is why the F16 GGUF holds 35.53B weights: the text model plus MTP, without vision.
The official Q4_K_M is the weakest file at its size
The official Q4_K_M has 84,709 downloads, the most of any Sage-A1.5 file. It was made without an importance matrix (imatrix). Two imatrix Q4_K_M files have the same size to within 1,056 bytes.
na0x2c6 measured each file against the BF16 logits on Japanese and English Wikipedia text. These are community measurements, from the na0x2c6 card:
| File | GiB | Mean KLD (ja / en) | Same top token (ja / en) |
|---|---|---|---|
| Q4_K_M, official | 20.22 | 0.0390 / 0.0378 | 90.9% / 92.0% |
| Q4_K_M, imatrix | 20.22 | 0.0255 / 0.0227 | 92.9% / 93.8% |
| IQ4_XS-exps | 18.00 | 0.0304 / 0.0264 | 92.1% / 93.3% |
| IQ3_S-exps-down4 | 16.28 | 0.0441 / 0.0370 | 90.2% / 92.3% |
Same memory, closer to the original. The imatrix file costs no extra VRAM. Even the 18.00 GiB IQ4_XS-exps file measured closer to BF16 than the official Q4_K_M.
Why the Context Costs So Little
Only the 10 full-attention layers keep a cache that grows with context. The 30 linear-attention layers keep a fixed state.
1. The full-attention cache. Config.json gives num_key_value_heads 2 and head_dim 256. layer_types lists 10 full_attention layers, one in every 4.
f16: 2 (K and V) x 2 heads x 256 x 2 bytes x 10 layers = 20,480 bytes per token
32,768 tokens = 0.625 GiB
65,536 tokens = 1.25 GiB
131,072 tokens = 2.5 GiB
262,144 tokens = 5.0 GiB
q8_0 cache (34 bytes per 32 values): 10,880 bytes per token = 2.66 GiB at 262K2. The linear-attention state. llama.cpp sizes it in llama-hparams.cpp. Config.json gives 16 key heads, 32 value heads, head size 128 and a conv kernel of 4. llama.cpp keeps this state in f32.
conv state: 3 x (4,096 + 2 x 16 x 128) x 4 bytes = 98,304 bytes per layer
recurrent state: 128 x 4,096 x 4 bytes = 2,097,152 bytes per layer
30 layers = 62.8 MiB, at any context length3. MTP adds one more layer. llama.cpp’s MTP PR says the MTP head runs with its own context and KV cache. By our arithmetic, that is one more attention layer: 2,048 bytes per token, or 0.5 GiB at 262K. That is our estimate, not a measured figure.
4. The full budget. Weights + cache + 0.06 GiB state + a compute buffer. We budget 1 GiB for the buffer. We did not measure it.
Our budget is conservative. prithivMLmods ran the 18.87 GiB MXFP4_MOE file at 8K context with a q8_0 K cache. Their card reports a 19.72 GiB peak. Our method gives about 20.0 GiB for that setup.
Fit Table by Hardware
GPU rows assume the whole card is usable, minus about 0.5 GiB for the driver. Mac rows use the default Metal limit. Every row is our arithmetic unless it says community-reported. Add 0.57-0.84 GiB to any row for the vision file.
| Hardware | Best file | Context | Total | Notes |
|---|---|---|---|---|
| 16GB GPU, all on GPU | Q2_K (12.34) | 64K f16 | ~14.7 GiB | 2-bit; quality cost |
| 16GB GPU + 64GB DDR5 | Q4_K_M (~20.2) | 131K | split | Experts in RAM; 166 tok/s at 4K (community) |
| 24GB GPU, all on GPU | Q4_K_M (20.22) | 64K f16 / 128K q8_0 | ~22.5 / ~22.6 GiB | The verdict |
| 24GB GPU, full context | i1-IQ4_XS (17.86) | 262K q8_0 | ~21.6 GiB | 4-bit experts |
| 24GB GPU, with vision | Q4_K_M + Q8_0 mmproj | 64K f16 | ~23.1 GiB | Tight |
| 32GB GPU (RTX 5090) | Q5_K_M (23.61) | 262K f16 | ~29.7 GiB | Full context, room for vision |
| 32GB GPU, more bits | Q6_K (27.20) | 128K f16 / 262K q8_0 | ~30.8 / ~30.9 GiB | Tight |
| 32GB Mac (24.96 GiB limit) | Q4_K_M or MLX OptiQ (20.62) | 128K f16 / 262K q8_0 | ~23.8 / ~23.9 GiB | Close the browser |
| 48GB Mac (37.44 GiB limit) | Q6_K (27.20) | 262K f16 | ~33.3 GiB | Q8_0 fits only at 32K (~36.9) |
| 64GB Mac (51.84 GiB limit) | Q8_0 (35.21) | 262K f16 | ~41.3 GiB | Room for vision and MTP |
| Strix Halo 64GB | Q5_K_M imatrix | 64K | — | 58-65 tok/s with MTP (community) |
The 24GB card is the clean tier. Q4_K_M + 1.25 GiB cache + 0.06 GiB state + 1 GiB buffer is 22.53 GiB. A q8_0 cache doubles that to 128K for about the same memory. A used RTX 3090 24GB runs it with no offload.
Full context on 24GB costs a size step. Q4_K_M with the full 262K f16 cache is about 26.3 GiB. With a q8_0 cache it is about 23.9 GiB, which leaves no room. Drop to i1-IQ4_XS (17.86 GiB) for the full context.
The 32GB card runs 5 bits at full context. Q5_K_M + 5.0 GiB cache + buffers is about 29.7 GiB. The vision file still fits.
Mac limits. llama.cpp prints Metal’s recommendedMaxWorkingSetSize at startup, in units of 10^6 bytes. We re-read three logs. A 32GB M4 Mac mini logs 26,800.60 MB (harborclerk #684), which is 24.96 GiB. A 48GB M4 Pro MacBook Pro logs 40,200.90 MB (llama.cpp #20141), which is 37.44 GiB. A 64GB M4 Pro Mac mini logs 55,662.79 MB (llama.cpp #20011), which is 51.84 GiB. You can raise the limit with sudo sysctl iogpu.wired_limit_mb=N, with N in MiB.
The 16GB route works because the experts are small. Each expert is 3 x 2,048 x 512 weights. A 16GB card holds the always-read weights, the cache and a share of the experts. The rest live in RAM. llama.cpp’s --moe-cache-mib keeps the most-used RAM experts in a VRAM cache.
How Fast Will It Be?
We found four owners who posted numbers. All are community-reported, from the Hugging Face pages linked in Sources.
| Setup | Prompt processing | Generation |
|---|---|---|
RTX 5080 16GB + 64GB DDR5-6400, Q4_K_M, llama.cpp b11834, --cpu-moe --moe-cache-mib 10300, MTP 2, at 4K | 3,185 tok/s | 166 tok/s |
| Same, at 40K | 3,730 tok/s | 146 tok/s |
| Same card, Strata fork (not llama.cpp), at 240K | 2,477 tok/s | 162 tok/s |
| Ryzen AI Max+ 392, 64GB, Q5_K_M imatrix, 64K, Vulkan, MTP 2 | not given | 61-65 tok/s thinking, 63-77 tok/s code |
| Same, no MTP | not given | 58-59 tok/s |
| RTX PRO 6000 96GB, MXFP4_MOE, 8K, MTP 2 | 94.1 tok/s | 270.4 tok/s |
MTP helps, but less than the headline. The Strix Halo owner measured a 9-12% speed-up while thinking and 13-34% while writing code. Draft acceptance was 58-88%, by task.
Bandwidth ceilings (derived). Generation reads the active weights once per token.
Active weights per token (our count): ~2.95B
Q4_K_M average: 21,713,463,520 bytes x 8 / 35.53B weights = 4.89 bits per weight
Bytes read per token: 2.95B x 4.89 / 8 = ~1.80 GB
RTX 5060 Ti 16GB 448 GB/s -> ~249 tok/s ceiling
RTX 3090 24GB 936 GB/s -> ~520 tok/s ceiling
RTX 5090 32GB 1,792 GB/s -> ~996 tok/s ceiling
M5 Pro 307 GB/s -> ~171 tok/s ceilingThese are ceilings, not predictions. Real decode is lower, because kernels, the router and the linear-attention state all cost time. The ceilings show one thing well: on a 3B-active model, the card that holds the file beats the card that is faster.
Software Support, Checked 26 October 2026
| Runtime | Status | Source |
|---|---|---|
| llama.cpp | The qwen35moe architecture is supported since PR #19468, merged 10 February 2026. First release b7990. The PR says it was tested with and without an mmproj file | GitHub |
| llama.cpp MTP | --spec-type draft-mtp since PR #22673, merged 16 May 2026, first release b9180. The PR says MTP works with vision input | GitHub |
| llama.cpp issue | Q6_K load and speed problem on a 16GB RTX 5060 Ti with 32GB RAM, build 11860, still open | HF discussion |
| Ollama | No sage-a1.5 library entry. The library has qwen3.6:35b (23-24GB), the same architecture. We found no report of Ollama running Sage’s GGUF | ollama.com |
| LM Studio | No Sage-A1.5 page. LM Studio lists qwen/qwen3.6-35b-a3b, tag qwen35moe, “Minimum system memory 20 GB”. We found no report for Sage | lmstudio.ai |
| MLX | mlx-lm has qwen3_5_moe text support since 12 February 2026 (PR #869). mlx-community’s OptiQ card says stock mlx-lm loads it text-only, and optiq serve adds images | GitHub, HF |
| vLLM, SGLang | The card gives serve commands for both, including MTP and a --language-model-only text mode. vLLM is “from the main branch” | model card |
Run the 24GB setup in llama.cpp:
llama-server -m Sage-A1.5.i1-Q4_K_M.gguf -ngl 99 -c 65536 -fa on --jinja \
--temp 1.0 --top-p 0.95 --top-k 20 --min-p 0 --presence-penalty 1.5For 128K on 24GB, add -c 131072 -ctk q8_0 -ctv q8_0. For images, add --mmproj Sage-A1.5-mmproj.gguf. For MTP, add --spec-type draft-mtp --spec-draft-n-max 2. On a 16GB card, add --cpu-moe, or --n-cpu-moe N and lower N until the GPU is full. Our —n-cpu-moe guide shows how.
Use the card’s sampling, especially the presence penalty. The mlx-community card tested its quant without presence_penalty 1.5. On two prompts, it reasoned past 16,384 tokens with no answer. With the card’s values, both finished. Many agent clients send no sampling values, so set them in the server.
Long thinking is a known problem. The Strix Halo owner saw 30,000 thinking tokens with no answer on a hard coding task. A 10,000-token reasoning budget closed the thinking block. A user of the official Q4_K_M also reports repetition loops on long coding prompts.
What to Buy
Prices as of October 2026, from our price reference. The read date is in brackets.
| Option | Price | What it runs |
|---|---|---|
| RTX 5060 Ti 16GB | $789 at B&H (24 September); $589-805 range (August) | Q4_K_M with experts in RAM |
| RTX 3090 24GB (used) | $1,541 average, 317 listings (resaleprices.com, 22 October) | Q4_K_M fully on the GPU, 64-128K |
| RTX 5090 32GB | from $6,987, 92 marketplace listings (GPU Poet, 23 October) | Q5_K_M, full 262K context |
| Mac mini M5 Pro 48GB | $2,299 (apple.com, 22 September) | Q6_K with full context |
| Mac mini M5 Pro 64GB | $2,699 (apple.com, 22 September) | Q8_0 with full context |
For most people, buy 24GB. A used RTX 3090 runs the 4-bit Sage-A1.5 with no offload and no tuning. It costs under a quarter of a marketplace 5090.
Already own a 16GB card? Do not replace it for Sage-A1.5. Add RAM and put the experts in it. The 166 tok/s report used 64GB of DDR5. If you are buying new at 16GB, the RTX 5060 Ti 16GB runs the 4-bit file with experts in RAM. Pair it with 64GB of system RAM, not 32GB.
Buy 32GB only for 5 bits at full context. The RTX 5090 32GB holds Q5_K_M, all 262K tokens and the vision file. Marketplace prices are far above its $1,999 list, so check the price first.
On a Mac, buy 48GB or more. A 32GB Mac runs 4 bits, but with almost no room left. The Mac mini M5 Pro has 307 GB/s of bandwidth. Its 48GB config holds Q6_K at full context, and its 64GB config holds Q8_0.
Honest Caveats
- We did not run the model. Every fit row is arithmetic from file sizes and config.json.
- Every speed number is one owner’s post. No two runs used the same file and settings.
- The 1 GiB compute buffer is a budget. One measured peak came in below our method. Check the buffer sizes in your
llama-serverlog. - The MTP cache cost is our estimate. We read it from the PR text, not from a log.
- The KLD comparison is one quantizer’s test. It covers short Wikipedia text, not agent work.
- The benchmark table is InternScience’s. Community reports are positive but mention long thinking and loops. Test the model on your own tasks.
Sources
- Sage-A1.5 model card: base models, modes, benchmark claims, sampling values, SGLang and vLLM commands, YaRN note
- config.json: layer_types, num_key_value_heads, head_dim, linear-attention heads, experts, moe_intermediate_size, shared_expert_intermediate_size, max_position_embeddings, vision_config
- Hugging Face API: creation date, parameter count, licence
- File trees for official Q4_K_M, Q6_K, Q8_0, F16, mradermacher i1, mradermacher static, na0x2c6, prithivMLmods MXFP4, mlx-community OptiQ, official FP8 and kvallespin NVFP4: byte sizes
- na0x2c6 GGUF card: KLD table, MTP layer kept, imatrix method
- prithivMLmods MXFP4 card: RTX PRO 6000 speed and 19.72 GiB peak
- mlx-community OptiQ card: ~3B active, file layout, presence-penalty test, mlx-lm text-only note
- Community threads: Sage discussion #4 (RTX 5080, llama.cpp and Strata), Sage discussion #2 (Strix Halo, MTP, long thinking), Q4_K_M discussion #2 (repetition loops), Q6_K discussion #3 (16GB card load problem)
- llama.cpp PR #19468, release b7990, PR #22673, release b9180,
src/llama-hparams.cpp(state sizes) andcommon/arg.cpp(--moe-cache-mib) - Metal working set logs: harborclerk #684, llama.cpp #20141, llama.cpp #20011
- Ollama library: qwen3.6 tags and LM Studio: qwen/qwen3.6-35b-a3b
- mlx-lm qwen3_5_moe.py history (PR #869)
- Bandwidth: Wikipedia GeForce RTX 50 series (448 and 1,792 GB/s) and GeForce RTX 30 series (936 GB/s); bus widths on nvidia.com; M5 Pro 307 GB/s from our price reference (apple.com)
- OpenClaw DC hardware price reference, last updated 26 October 2026: RTX 5060 Ti, RTX 3090, RTX 5090 and Mac mini rows
See Also
- Can I run Flint locally?: another new MoE that fits a 24GB card
- Edge0 35B-A3B on a 16GB Mac mini: another Qwen3.6-35B-A3B derivative, on 16GB
- Best local LLM for 24GB VRAM: what else to run on the 24GB tier
- Best local LLM for 32GB VRAM: the 32GB tier
- Best local LLM for the RTX 5060 Ti 16GB: the 16GB tier
- Best local LLM for MacBook: Mac memory limits by RAM size
- —n-cpu-moe explained: how to put experts in system RAM