Tokens Per Second Calculator

How fast will a local model write on your machine, and is that fast enough for what you do? Token generation is limited by memory bandwidth: every new token reads the model's active weights and its KV cache once. Pick your chip or card, a model and a context length. The estimate is scaled from llama.cpp's own benchmark of that device.

How the estimate works

bytes per token = file size x (active params / total params) + KV cache at your context
estimate        = reference t/s x 3.56 GiB / bytes per token
  reference t/s = llama.cpp tg128 for Llama 2 7B Q4_0 (3.56 GiB) on that device
ceiling         = memory bandwidth / bytes per token   (shown where the vendor figure was read)
MoE range       = 68% to 100% of the estimate     (one measured MoE run, below)

Worked example: dense 27B vs 35B-A3B at 4K context

DeviceLlama 2 7B Q4_0 (measured)Qwen3.8 27B UD-Q4_K_MQwen3.6 35B-A3B UD-Q4_K_M
M424.11 t/sabout 5.5 t/s32.3 to 47.7 t/s
M4 Pro (20-core GPU)50.74 t/sabout 11.5 t/s67.9 to 100 t/s
M4 Max (40-core GPU)83.06 t/sabout 18.9 t/s111 to 164 t/s
M5 Max (40-core GPU)119.92 t/sabout 27.2 t/s161 to 237 t/s
RTX 3090 24GB158.16 t/sabout 35.9 t/s212 to 313 t/s
RTX 5090 32GB290.02 t/sabout 65.9 t/s388 to 574 t/s

Fit is not checked in this table: the 27B and the 35B-A3B files do not fit every device listed. The tool above checks fit.

Measure your own

Ran a model on your machine? Add a Mac result to llama.cpp's table, or reply on the tokens-per-second guide.

Questions

How many tokens per second is good for a local LLM?

It depends on the task. Community floors: about 5 tokens per second for chat you read, 20+ for interactive coding, 40+ for agent loops, and 1 or more for overnight batch jobs nobody watches. This calculator checks your estimate against the floor for the task you pick.

How do I estimate tokens per second before I download a model?

Divide your memory bandwidth by the bytes the model reads per token. A dense model reads its whole file for every token; an MoE model reads only its active share. This page anchors that line on llama.cpp's own benchmark of each chip and card (Llama 2 7B Q4_0, 3.56 GiB), so the estimate already includes how far below the bandwidth ceiling that device runs.

Why is a dense 27B model so slow on a base M4 Mac?

Because it reads all of its weights for every token. Qwen3.8 27B at UD-Q4_K_M is 15.4 GiB, and the M4 has 120 GB/s, so the estimate is about 5.5 tokens per second. Qwen3.6 35B-A3B reads about 3B of its 35.95B parameters per token, so on the same chip the estimate is 32.3 to 47.7.

Why does my model get slower as the conversation gets longer?

The KV cache is read on every token, and it grows with the context. At 4K it is small; at 128K it can be several GiB, which adds to the bytes per token. Set the context to the length you actually run to see the slowdown.

Is 20 tokens per second good?

For coding with a model you watch, 20 is the community floor. For chat it is comfortable (the floor is about 5). For agent loops it is slow: the agent writes plans and tool calls you never read, and the floor people quote is 40+.

Related