Speculative Decoding Made My Local LLM Slower (August 2026): Why the Draft Model Backfires and How to Tell

Bottom Line (August 2026)

• It is a bet, not a switch. Speculative decoding pays off in proportion to how guessable your output is. Rejected drafts are wasted compute, so low-acceptance workloads run net slower.
• Workload dominates everything else. Community benchmarks on the same hardware and model pair reported over 4x on a high-draftability refactoring prompt and about 1.3x on “tell me the rules of chess.” Same setup, same flags.
• Apple Silicon is frequently a net loss. Metal-backend testing reports draft-model speculative decoding at roughly -11% to -24%. Unified memory means the draft is not running on spare capacity.
• Same-device contention is a real failure mode, not a tuning problem. A Vulkan bug report has draft per-token time exploding when both models sit on the same iGPU, wiping out any gain.
• Try n-gram drafting first. --spec-type ngram-simple costs no VRAM and no second model. On repetitive output it is most of the win for none of the setup.
• Measure with and without, on your own prompts. Read the acceptance stats llama.cpp prints. If they are low and tokens/sec is flat, the correct action is to remove the draft model.

Why a Draft Model Can Cost You Time

The mechanic is simple. Generating one token at a time is memory-bandwidth bound — you drag the whole weight set through memory for a single token. But verifying n tokens at once is closer to prompt processing, which is compute bound and much more efficient per token.

So a small draft model proposes several tokens ahead. The big model runs one forward pass over all of them and checks which ones it would have produced itself. Everything up to the first disagreement is accepted for free. Everything after the disagreement is thrown away.

That last sentence is the whole story. The throwaway is not free. You paid the draft model’s compute to produce those tokens, and you paid to verify them. When the draft is usually right, that cost is trivially recovered. When the draft is usually wrong, you are running two models to get the output of one.

This is why the same config can be a 4x win and a 10% loss depending on nothing but what you asked it to write.

Diagnostic Flow

Work down this table. The first two rows explain the large majority of “it got slower” reports.

Cause 1 — Your Prompts Are Not Draftable

This is the cause people least want to hear, because there is no flag that fixes it.

Community benchmarks with a Qwen coder target drafted by a 0.6B same-family draft reported, on an RTX 5000 Ada, roughly 80 tok/s drafted versus 18 tok/s undrafted on a refactoring prompt — over 4x. The same setup on a low-draftability prompt like explaining the rules of chess came in around 1.3x. On an M1 Ultra 64GB the same refactoring prompt gave just under 2x (roughly 24 versus 14 tok/s), and the chess prompt gave about 1x — no gain at all.

Those are community-reported numbers on specific hardware, and yours will differ. What transfers is the shape: the spread between best and worst case on identical configuration is enormous, and it is driven entirely by the content.

Refactoring wins because most of the output is a near-copy of the input. The draft model does not need to be smart, it needs to be right, and “reproduce this line with one identifier changed” is something a 0.6B model gets right constantly.

Open-ended prose loses because every token is a genuine choice among many plausible continuations. The small model diverges early, the draft gets rejected at token two, and you did the work for nothing.

Practical consequence: if you run one server for both coding agents and chat, speculative decoding is not a global setting you tune once. It is a per-workload decision.

Cause 2 — Apple Silicon and the Shared-Bandwidth Problem

Reports on llama.cpp with the Metal backend consistently put draft-model speculative decoding at a net loss, in the neighborhood of -11% to -24%.

The reason is structural. On a discrete-GPU desktop, the draft model is small enough that running it uses capacity the target model was not saturating anyway. On unified memory, the draft model and the target model are pulling from the same pool at the same bandwidth ceiling. There is no spare capacity for the draft to run in, so its cost lands directly on your token time — and then you still throw away the rejected tokens.

If you are on an M-series Mac and speculative decoding got you nothing or hurt, you have not misconfigured anything. Try the n-gram approach below instead, and if that also does nothing, this optimization is not for your machine.

Cause 3 — Same-Device Contention

There is a sharper version of the bandwidth problem that shows up as a catastrophic slowdown rather than a mild one.

An open llama.cpp issue describes the Vulkan backend on a unified-memory iGPU: with both models loaded on the same device, the draft model’s per-token generation time balloons by orders of magnitude versus running it standalone, so the drafted config is no faster than — or slower than — the undrafted baseline.

The tell is the magnitude. Low draftability costs you 10-25%. This costs you almost everything. If you added a small draft model and your throughput fell off a cliff, check whether both models landed on the same constrained device before you go tuning draft lengths.

llama.cpp lets you place the draft separately:

-devd    # or --spec-draft-device 
-ngld all        # or --spec-draft-ngl all

On a two-GPU box, putting the draft on the second card removes the contention entirely. On a single-device machine there is nowhere to move it to, and the answer is to stop using a draft model.

Cause 4 — Vocabulary Mismatch

For plain draft-model speculation, the draft and target must share a tokenizer. The draft emits token IDs; the target verifies those IDs. If ID 4817 means different things to the two models, verification is meaningless.

This is why the working pairs you see in the wild are always same-family: a 0.5B or 0.6B draft against a 7B-to-35B target from the same release. Grabbing whatever small model you already have downloaded is the usual mistake.

The exception is EAGLE-3-style drafts, which use a reduced draft vocabulary with their own lm_head mapped back through a table. llama.cpp’s conversion step handles this — you pass --target-model-dir when converting the EAGLE-3 checkpoint so the draft inherits the target’s tokenizer and the layer indices it needs to read.

The Flags (and a Naming Warning)

llama.cpp renamed this family of arguments. Current builds use --spec-* names; older builds and most tutorials you will find use the earlier --draft-* and --model-draft spellings. Check llama-server --help on your actual binary — this is the single most common reason a copied command line errors out with an unrecognized argument.

Current naming:

-md,   --spec-draft-model FNAME    # draft model file
       --spec-draft-n-max N        # tokens to draft per step (default 3)
       --spec-draft-n-min N        # minimum draft tokens (default 0)
       --spec-draft-p-min P        # min probability for greedy drafting (default 0.00)
-ngld, --spec-draft-ngl N          # draft GPU layers; accepts a number, 'auto', or 'all'
-devd, --spec-draft-device LIST    # comma-separated devices for the draft

And the strategy selector, which takes comma-separated values:

--spec-type draft-simple      # classic small draft model
--spec-type draft-eagle3      # EAGLE-3 draft, reads target hidden states
--spec-type ngram-simple      # pattern-based, no second model
--spec-type ngram-mod,ngram-map-k4v

A working draft-model server looks like this:

llama-server \
  -m ./qwen3.6-coder-30b-Q5_K_M.gguf \
  -md ./qwen3.6-coder-0.6b-Q4_0.gguf \
  --spec-type draft-simple \
  --spec-draft-n-max 5 \
  -ngl all -ngld all \
  --flash-attn on \
  -c 32768 \
  --host 127.0.0.1 --port 8080

Substitute your own paths and quants. --spec-draft-n-max is worth a short sweep: drafting further ahead wins more when acceptance is high and wastes more when it is low, so the optimum tracks your workload. Start at the default and try 5 and 8.

The Option Most People Skip: N-Gram Drafting

Pattern-based drafting needs no second model at all. It searches the existing token history for matching patterns and drafts the continuation from what it finds.

llama-server -m ./model.gguf --spec-type ngram-simple -ngl all -c 32768

Zero extra VRAM, zero extra weights, nothing to download, no tokenizer compatibility question. On a memory-constrained card this should be the first thing you try, because the draft-model version of this optimization can only help you after you have already paid for the draft in VRAM.

The tradeoff is that n-gram drafting only works when your output repeats things already in the context. Editing a file that is in your prompt: excellent. Answering a question from scratch: close to nothing. Which is, again, the same draftability story from a different angle.

How to Decide, in Ten Minutes

1. Benchmark the baseline. Run your real prompt with no speculation and record tokens/sec. Not a synthetic benchmark — the prompt you actually send.
2. Add the draft, rerun the same prompt. Same context size, same quant, same everything else.
3. Read the acceptance stats. llama.cpp reports accepted-token counts and rates at the end of a run. Low acceptance with flat throughput is your answer.
4. Repeat for your second workload. If you use the same server for coding and chat, benchmark both. They will disagree.
5. Watch VRAM while you do it. If loading the draft pushed you into a spill, you are measuring the spill, not the speculation. watch -n1 nvidia-smi --query-gpu=memory.used,memory.total --format=csv

If the answer is “no faster,” remove it. There is no partial credit here and no configuration that rescues a fundamentally unguessable workload.

Common Mistakes

1. Treating it as a global speed setting. It is workload-specific. The same config is a 4x win and a 1x wash depending on the prompt.
2. Using an unrelated small model as the draft. Different tokenizer, no speculation. Use a same-family draft.
3. Copying flag names from a 2025 tutorial. The arguments were renamed to --spec-*. Read your binary’s --help.
4. Not measuring the baseline first. Without an undrafted number on the same prompt you cannot tell a win from a loss.
5. Ignoring the VRAM the draft consumes. On a full card, adding a draft model can trigger the silent spill that costs far more than speculation saves.
6. Skipping n-gram drafting. It is free, it needs no download, and on repetitive output it captures much of the available win.

Related Guides

• llama.cpp MoE offload flags explained — the sweep method, and why the VRAM cliff is silent
• Why local LLMs are slow even when they fit — the spill failure mode that masquerades as everything else
• KV cache quantization: q8_0 vs q4_0 vs f16 — the other memory knob people reach for and misread
• Usable tokens per second by task — how fast you actually need to be before optimizing further
• Ollama vs llama.cpp — when the extra control is worth leaving Ollama