llama.cpp MoE Offload Flags Explained (July 2026): Run 35B Models Fast on a Mid-Range GPU

Bottom Line (July 2026)

• The trick: MoE models (Qwen 3.6 35B-A3B, Gemma 4 26B-A4B, Laguna XS 2.1) store tens of billions of parameters but activate only a few billion per token. Put the expert weights in system RAM, keep attention on the GPU.
• The flag: --n-cpu-moe N moves the MoE feed-forward weights of the first N layers to CPU. Keep -ngl at all — you are not offloading layers, you are offloading experts.
• Community receipts: an RTX 3060 12GB with 32GB DDR5 reported ~51-53 tok/s on Qwen3.6-35B IQ4_NL at 64K context. An RTX 4080 owner reported 60 tok/s on the same class of model. A 16GB RX 9060 XT user forces 24 expert layers to CPU.
• Tune by sweeping, not by copying. Lower --n-cpu-moe until VRAM spills, then back off one step. A 5090 owner measured 54 t/s at 20, 64.5 at 16, 69.4 at 12, and 27.5 one step further — the cliff is silent.
• f16 KV cache is faster than q8_0 when it fits. Quantized KV is a memory feature, not a speed feature.
• Numbers are machine-specific. Quant, RAM speed, context length, and CPU all move them. The method transfers; the numbers do not.

Why This Works at All

A dense 35B model has to move all 35B parameters through memory for every single token. That is why it is slow when it does not fit on the card.

A Mixture-of-Experts model at 35B total with 3B active is a different shape. Each token routes through a small subset of the experts. Most of those weights sit idle for most tokens. They still have to exist somewhere, but they do not have to be fast.

That is the whole insight. The parts that get touched every token — attention, embeddings, the KV cache — are small and belong on the GPU. The expert stacks are large and rarely touched, so they can live in system RAM at a fraction of the bandwidth without dominating your token time.

The old way to handle a model that did not fit was to lower -ngl and push entire layers to the CPU. That hurts, because those layers include attention. The MoE offload flags split the layer instead: attention stays on the card, experts go to RAM.

One commenter put it bluntly after finding this: “I thought my RTX4080 couldnt handle qwen35b. Now I run it with 60 t/s. You just saved me 5k$” (community-reported).

The Flags, One at a Time

-ngl (--n-gpu-layers)

How many transformer layers to place on the GPU. With MoE offload you generally want all of them:

-ngl all

This is the part people get wrong. The instinct from dense models is to dial -ngl down until it fits. With --n-cpu-moe available, keep -ngl maxed and control memory with the MoE flag instead. You want attention on the GPU for every layer.

--n-cpu-moe N

The main flag. Keeps the MoE feed-forward (expert) weights for the first N layers in system RAM.

--n-cpu-moe 25

Higher N means less VRAM used and more CPU work. Lower N means more experts on the GPU and faster generation, right up until you run out of VRAM.

The sweep method. Do not copy someone else’s number. Start high — a value you are sure fits — and step down, measuring tokens/sec at each stop:

for n in 32 28 24 20 16 12; do
  echo "--- n-cpu-moe $n ---"
  llama-bench -m ./qwen3.6-35b-a3b-IQ4_NL.gguf -ngl 999 --n-cpu-moe $n
done

Speed climbs as N drops, then falls off a cliff. One 5090 owner reported this sweep (community numbers): 20 gave 54 t/s, 16 gave 64.5 t/s, 12 gave 69.4 t/s — and one step below 12 collapsed to 27.5 t/s. That collapse is a VRAM spill. On many driver setups it does not throw an error; the allocation silently falls back to system memory and everything gets slow.

So the rule is: find the cliff, then back off one step. That is your setting until you change model, quant, or context size.

--flash-attn on

Fused attention kernel. Lower VRAM use for the KV cache and faster attention, especially at long context. On supported hardware this is close to free — turn it on and leave it on.

--flash-attn on

If you are running long context on a small card, this is often what makes 64K viable at all.

--no-mmap and --mlock

By default llama.cpp memory-maps the GGUF file. That is elegant when the whole model is on the GPU and irrelevant when it is not — but with a large CPU-resident expert set, mapping means the weights get counted in page cache and in your resident set, and the kernel starts making eviction decisions you did not ask for.

--no-mmap

One community report measured system RAM use dropping from 98% to 71% at the same generation speed after adding --no-mmap. Ninety-eight percent RAM is the zone where one background process pushes you into swap and the model appears to “randomly” get slow.

--mlock is the other side of the same problem: it pins pages so the OS cannot swap them out. Useful if you have headroom and something else on the machine keeps evicting your weights. Not useful if you are already tight on RAM — it will just fail or starve everything else.

-c (context size)

-c 65536

The KV cache grows with context, and it lives on the GPU. Every token of context you request is VRAM you cannot give to experts. If you set -c 131072 and then sweep --n-cpu-moe, you will find a much worse optimum than if you had asked for 32K.

Set context to what you actually need first, then sweep. Doing it in the other order gives you a tuned config for a workload you do not run.

KV cache quantization

You can store the KV cache at lower precision:

--cache-type-k q8_0 --cache-type-v q8_0

Here is the finding that surprises people: f16 KV cache is faster than q8_0 when it fits. Community testing reports this consistently. Quantizing the cache adds conversion work on every token, so you pay compute to save memory.

Quantized KV is a memory feature, not a speed feature. Reach for it when f16 does not fit and your alternative is either a shorter context window or a VRAM spill — both of which are worse. If f16 fits at the context you need, use f16.

-b / -ub (batch and micro-batch)

-b 1024 -ub 256

Batch buffers are allocated on the GPU. Shrinking them frees VRAM that you can hand back to experts via a lower --n-cpu-moe. The cost is slower prompt processing — generation speed is barely affected.

That trade is often worth it: prompt processing happens once per turn, generation happens for every token. If dropping -ub to 256 lets you move four more expert layers onto the card, take it.

MTP / speculative decoding

Where the model ships multi-token prediction weights or you have a compatible draft model, speculative decoding can add real throughput on top of everything above:

--spec-type draft-mtp

Support is model-specific and moves fast. Check whether your GGUF actually includes MTP weights before assuming this does anything — if it does not, the flag is a no-op at best.

Worked Configs

These are starting points to sweep from, not tuned answers. Substitute your own model path and quant.

12GB card (RTX 3060 12GB, 32GB system RAM)

llama-server \
  -m ./qwen3.6-35b-a3b-IQ4_NL.gguf \
  -ngl all \
  --n-cpu-moe 25 \
  --flash-attn on \
  --no-mmap \
  -c 65536 \
  --host 127.0.0.1 --port 8080

This is close to a community-reported working config: an RTX 3060 12GB with 32GB of DDR5 running Qwen3.6-35B at IQ4_NL, roughly 51-53 tok/sec at 64K context. On 12GB the KV cache is a large share of your budget, which is why --flash-attn matters so much here.

16GB card (RTX 4080, RX 9060 XT, 4060 Ti 16GB)

llama-server \
  -m ./qwen3.6-35b-a3b-IQ4_NL.gguf \
  -ngl all \
  --n-cpu-moe 20 \
  --flash-attn on \
  --no-mmap \
  -c 32768 \
  -b 1024 -ub 256 \
  --host 127.0.0.1 --port 8080

Four more GB buys you roughly a handful of expert layers back on the card. A 16GB RX 9060 XT owner reports forcing 24 expert layers to CPU as their working setting; an RTX 4080 owner reports 60 tok/sec on this model class. Both are community-reported, and both are in the neighborhood you should sweep around — start at 24 and walk down.

Sanity check before you tune

Watch VRAM while the model loads and during a long generation. If used VRAM sits just under your card’s capacity and speed is good, you are at a healthy setting. If it pegs at capacity and generation is a third of what it was one step ago, you found the cliff.

watch -n1 nvidia-smi --query-gpu=memory.used,memory.total,utilization.gpu --format=csv

Common Mistakes

1. Lowering -ngl instead of raising --n-cpu-moe. Dropping -ngl sends attention to the CPU, which is the expensive part. Keep -ngl at all and control memory with the MoE flag.
2. Copying someone’s --n-cpu-moe number from a forum. Their quant, RAM speed, and context are not yours. The sweep takes ten minutes and beats any number you can borrow.
3. Quantizing the KV cache for speed. It is a memory feature. f16 is faster when it fits.
4. Tuning at a context size you do not use. Set -c to your real workload first, then sweep.
5. Ignoring a silent spill. A sudden 2-3x slowdown with no error is almost always VRAM overflowing into system memory. Step --n-cpu-moe back up one.
6. Leaving mmap on with a large CPU-resident expert set. --no-mmap cost one user nothing in speed and gave back a quarter of their system RAM.

Related Guides

• Why local LLMs are slow even when they fit — the spill failure mode in detail
• Ollama vs llama.cpp — when the extra control is worth leaving Ollama
• Best local LLM for the RTX 3060 12GB and RTX 4060 Ti 16GB — model picks for the cards this technique helps most
• Can you run a 160B MoE on 8GB VRAM? — how far the offload trick actually stretches
• Laguna XS 2.1 on 24GB vs 32GB — the same MoE math, applied to a card-buying decision