Why Local LLMs Are Slow Even When They Fit

The mistake: treating memory fit as the whole problem

Most local LLM advice starts with one question: does the model fit?

That is the right first question, but it is not enough. A 27B or 35B model can fit in Apple unified memory, a 24GB GPU, or a 32GB GPU and still feel slow because the model is not just a static file sitting in memory.

Every response has at least two very different phases:

1. Prefill: the runtime processes your prompt and builds the KV cache.
2. Decode: the runtime generates new tokens one at a time using that cache.

If you paste a whole repo, a giant chat history, or a 100K-token document, prefill dominates. If you ask a short question after the cache is warm, decode dominates.

The practical bottleneck checklist

Apple Silicon: unified memory is not magic

Apple Silicon makes local inference easy because system RAM and GPU memory are unified. That does not mean every model runs quickly.

Dense models still move a lot of weights per generated token. If a 27B dense model fits but runs at disappointing tok/sec, the bottleneck may be memory bandwidth or runtime implementation, not raw RAM capacity.

Test the same prompt through:

• MLX or MLX-LM
• llama.cpp directly
• Ollama
• LM Studio

Use identical context length and quantization. Otherwise the comparison is not meaningful.

MoE can be faster, but it is not automatically better

Mixture-of-Experts models activate only part of the model per token. That can improve decode speed because active parameters are lower than total parameters.

The tradeoff:

• MoE models can be faster at generation.
• Router and expert scheduling can add overhead.
• Tool-calling reliability may differ from the dense model.
• Runtime support matters a lot.

For OpenClaw agent work, a slightly slower dense model with reliable tool calls can beat a faster MoE that drifts, loops, or emits malformed JSON.

Quantization can break structure before it breaks “reasoning”

People often say Q4 or Q6 is “basically fine.” That is directionally true for many large models and casual chat. It is less true for strict structured output.

For OpenClaw, the failure mode is not always a bad answer. It is often:

• missing braces
• unquoted enum values
• extra prose inside JSON
• a tool call that looks plausible but does not validate
• a confirmation message for a write that never happened

For small models, test Q8 before trusting Q4/Q6 for tool calls. Pair that with schema-constrained output when the runtime supports it. Ollama documents both tool calling and structured outputs.

The fastest fix order

1. Reduce context length. Try 8K or 32K before assuming the model is bad.
2. Confirm full GPU residency. Partial offload can turn a good GPU into a PCIe bottleneck.
3. Separate prefill and decode. Benchmark first-token latency and tok/sec separately.
4. Test another runtime. Runtime choice can matter as much as the model.
5. Try an MoE for speed. Keep a dense fallback for tool-call reliability.
6. Use the estimator. It will flag whether your issue is memory fit, context, runtime, or experimental architecture.

When a model “fits” but should still be rejected

Reject the setup if:

• it needs swap for your real context length
• it only works at 3-bit when you need reliable code or JSON
• it has great chat speed but fails tool validation
• the first-token delay is too long for the workflow
• a smaller model with stronger scaffolding completes the job faster

For a local assistant or OpenClaw agent, the winning setup is not the largest model you can launch. It is the largest model that remains reliable under the exact workflow you run every day.

Next steps

• Run the Local LLM Fit and Speed Estimator
• Make local LLM tool calls reliable
• Compare AMD R9700 vs RTX 3090 for local LLMs
• Can you run a 160GB MoE on 8GB VRAM?