How fast will a local model write on your machine, and is that fast enough for what you do? Token generation is limited by memory bandwidth: every new token reads the model's active weights and its KV cache once. Pick your chip or card, a model and a context length. The estimate is scaled from llama.cpp's own benchmark of that device.
bytes per token = file size x (active params / total params) + KV cache at your context estimate = reference t/s x 3.56 GiB / bytes per token reference t/s = llama.cpp tg128 for Llama 2 7B Q4_0 (3.56 GiB) on that device ceiling = memory bandwidth / bytes per token (shown where the vendor figure was read) MoE range = 68% to 100% of the estimate (one measured MoE run, below)
eval rate: 33.96 tokens/s on qwen3.5:35b-a3b-coding-nvfp4 (22 GB on ollama.com) in Ollama's MLX engine. That is 68% of what this method predicts for it. MoE models route each token through different experts, which costs extra time, so the page shows a range for them. One community run is thin evidence; send yours below.config.json, the same math as the calculator.| Device | Llama 2 7B Q4_0 (measured) | Qwen3.8 27B UD-Q4_K_M | Qwen3.6 35B-A3B UD-Q4_K_M |
|---|---|---|---|
| M4 | 24.11 t/s | about 5.5 t/s | 32.3 to 47.7 t/s |
| M4 Pro (20-core GPU) | 50.74 t/s | about 11.5 t/s | 67.9 to 100 t/s |
| M4 Max (40-core GPU) | 83.06 t/s | about 18.9 t/s | 111 to 164 t/s |
| M5 Max (40-core GPU) | 119.92 t/s | about 27.2 t/s | 161 to 237 t/s |
| RTX 3090 24GB | 158.16 t/s | about 35.9 t/s | 212 to 313 t/s |
| RTX 5090 32GB | 290.02 t/s | about 65.9 t/s | 388 to 574 t/s |
Fit is not checked in this table: the 27B and the 35B-A3B files do not fit every device listed. The tool above checks fit.
ollama run <model> --verbose. eval rate is the writing speed this page estimates; prompt eval rate is prefill.llama-bench -m model.gguf prints pp512 (prefill) and tg128 (writing), the same test as the reference rows.Ran a model on your machine? Add a Mac result to llama.cpp's table, or reply on the tokens-per-second guide.
It depends on the task. Community floors: about 5 tokens per second for chat you read, 20+ for interactive coding, 40+ for agent loops, and 1 or more for overnight batch jobs nobody watches. This calculator checks your estimate against the floor for the task you pick.
Divide your memory bandwidth by the bytes the model reads per token. A dense model reads its whole file for every token; an MoE model reads only its active share. This page anchors that line on llama.cpp's own benchmark of each chip and card (Llama 2 7B Q4_0, 3.56 GiB), so the estimate already includes how far below the bandwidth ceiling that device runs.
Because it reads all of its weights for every token. Qwen3.8 27B at UD-Q4_K_M is 15.4 GiB, and the M4 has 120 GB/s, so the estimate is about 5.5 tokens per second. Qwen3.6 35B-A3B reads about 3B of its 35.95B parameters per token, so on the same chip the estimate is 32.3 to 47.7.
The KV cache is read on every token, and it grows with the context. At 4K it is small; at 128K it can be several GiB, which adds to the bytes per token. Set the context to the length you actually run to see the slowdown.
For coding with a model you watch, 20 is the community floor. For chat it is comfortable (the floor is about 5). For agent loops it is slow: the agent writes plans and tool calls you never read, and the floor people quote is 40+.