How Many Tokens Per Second Do You Actually Need? Usable Speed by Task (July 2026)

Bottom Line (July 2026)

• There is no single number. The community floor votes range from 1–2 t/s to 90 t/s, and the spread is task, not disagreement.
• Chat you read: ~5–10 t/s. One frequently cited floor is “at least human reading speed, around 5 tokens per second.” Another common line: “10 t/s is my lower limit on usability.”
• Interactive coding: 20+ t/s. You are waiting on whole functions, not reading word by word.
• Agentic loops: 40+ t/s. Agents generate reasoning and tool calls you never read, so every hidden turn multiplies wall-clock time.
• Overnight batch: anything. Users run 1–2 t/s happily when nobody is watching the stream.
• Every published t/s is a shallow-context number. Community measurement puts throughput roughly halving by 30K+ tokens of context.
• Prefill and decode are different numbers. A 20K-token agent system prompt is why “2+2” can take 20 seconds on fast hardware.
• Thinking models burn 5–30K reasoning tokens before the answer, so effective speed is far below the decode figure.
• MoE models decode at active-param speed. 33B total with 3B active runs like a small model.

The Thread That Has No Answer

When someone asks “how many tokens per second do I need,” the replies split hard, and the useful thing is that each person is describing their own workflow rather than a benchmark. Verbatim, the recurring positions:

“For useful work, we would want at least human reading speed, around 5 tokens per second”

“10tps is my lower limit on usability”

“To be generally useful I prefer 50-90 tps… Anything below 10 tps is not very useful to me”

And quietly, the people running summarization or document processing overnight report living comfortably at 1–2 t/s, because the output lands in a file they read in the morning.

None of these are wrong. The 5 t/s answer is about reading. The 50–90 answer is about not waiting. The 1–2 answer is about not caring. Before you buy a card or pick a quant, decide which of those you are.

Usable Speed by Task

Numbers are community-attributed floors and approximate comfort bands, not measurements from a single rig.

Autocomplete is the one that breaks the framing. A completion is 20 or 30 tokens. Throughput barely matters; what matters is how fast the first token arrives. A model doing 120 t/s with 400 ms of startup latency feels worse than one doing 40 t/s that starts instantly, because you have already typed the line yourself. Optimize small models and warm loads there, not raw t/s.

Agentic loops are the other one. In chat, one prompt produces one visible answer. In an agent loop, one prompt produces a plan, a tool call, a tool result read-back, a revision, another tool call, and eventually an answer. You are paying decode time for every one of those, and you see none of them. At 10 t/s a task with eight hidden turns is a coffee break. That is why the agent floor sits so far above the chat floor even though the model is identical.

Four Reasons the Advertised Number Lies

1. Every published t/s is a shallow-context number

Benchmarks measure on a fresh prompt with a nearly empty KV cache. Attention cost grows as the window fills, so throughput decays through a conversation. Community measurement puts it at roughly half by the time you are 30K+ tokens deep — one 12GB owner reports settling around 30 t/s past ~36K on a model that benchmarks far higher cold.

For agent work that decayed number is the real one, and almost nobody publishes it. Read every advertised figure as a best case at turn one. More on this failure mode in context window traps for local agents.

2. Prefill and decode are two different speeds

Prefill (prompt processing) is the model reading your input. Decode is it writing the output. They run at wildly different rates and only decode gets quoted.

This is invisible in chat, where your prompt is a sentence. It is brutal in an agent, where the prompt is a long behavioral system prompt plus a JSON schema for every registered tool — commonly 15–20K tokens before your message. A user on a 48GB M4 MacBook reported asking “2+2” and waiting roughly 20 seconds. The model spent that time reading its own instructions and about 50 milliseconds answering.

The fix is prompt caching or fewer registered tools. A faster model does nothing, because decode was never the slow part.

3. Thinking models spend your budget before the answer starts

Reasoning models emit a long internal chain before the visible response — commonly in the 5–30K token range depending on the model and the difficulty of the question. All of it is decoded at the same t/s.

So a model doing 40 t/s that thinks for 8,000 tokens makes you wait 200 seconds before the first useful word. Its effective speed on that question is a fraction of its decode rate. When you compare a thinking model against a non-thinking one, compare time-to-useful-answer, not t/s — the t/s column will actively mislead you.

4. MoE models are faster than their size implies

A Mixture-of-Experts model routes each token through a small subset of its parameters. A 33B-total model with 3B active decodes at roughly 3B speed while occupying 33B of memory. Total parameters set the memory bill; active parameters set the speed.

The practical consequence: do not skip a model because the parameter count sounds slow on your card. Check the active count. This asymmetry is also the single most effective lever when you are below your floor — see the 24GB vs 32GB breakdown for one worked example.

Measure Your Own

Do not argue from benchmarks. Get your number, on your hardware, at your context depth.

Ollama prints the rates with --verbose:

ollama run qwen3.5:32b --verbose "write a python function that parses an ISO 8601 timestamp"

Two lines matter in the output: prompt eval rate is your prefill speed, and eval rate is your decode speed — the number people mean when they say tokens per second.

llama.cpp prints timings after every generation, from both the CLI and the server log, separating prompt eval from eval.

Measure twice. Once on a short prompt, once on a realistic one — paste in a few thousand tokens of the code or documents you actually work with. The gap between those two runs is what you will feel day to day, and it is the number no benchmark gives you.

If You Are Below Your Floor

In rough order of how much speed they buy per unit of pain:

Drop the quant. Going from Q8 to Q4_K_M roughly halves the bytes read per token, and decode on consumer hardware is memory-bandwidth-bound, so this is close to a direct speed multiplier. Quality loss at Q4_K_M is modest for most work.

Switch to an MoE model. The biggest single jump available, per the point above. A comparable-quality MoE with a small active-parameter count can move you from single digits to comfortable.

Check for a KV cache spill. If a model was fast and suddenly crawls, part of it is running from system RAM. That is a multiple-times slowdown, not a percentage, and the fix is dropping the context window one notch rather than buying anything. Details in why local LLMs are slow even when they fit.

Tune your offload flags. On MoE models specifically, choosing which tensors stay on GPU matters more than the blunt layer count — the flags and what they do are in llama.cpp MoE offload flags explained.

Trim the agent’s tool set. This does nothing for decode but cuts prefill directly, which is often what you are actually feeling.

Then, and only then, buy hardware. Decode speed tracks memory bandwidth far more than it tracks compute. A card with more VRAM at similar bandwidth buys you context, not tokens per second — worth knowing before you assume an upgrade fixes a speed complaint.

Related Guides

• Why is my local LLM so slow — the diagnostic walkthrough
• Why local LLMs are slow even when they fit — the spill-to-RAM failure mode
• Context window traps for local agents — why speed decays as the window fills
• llama.cpp MoE offload flags explained — getting speed back on big MoE models
• Ollama crash course for local agents — the runtime settings that matter
• RTX 5090 vs 4090 vs used 3090 — if hardware really is the answer