vLLM vs llama.cpp vs SGLang for a Single User: Does It Matter?
For one user at a time, vLLM, llama.cpp, and SGLang produce tokens at a comparable rate. The 44x differences you read about only appear under concurrency. Here is the honest breakdown.
Short answer
At batch size 1 — one request at a time — vLLM and llama.cpp roughly tie. Red Hat benchmarked both and found they “produce tokens at a comparable rate” for a single request.
Ready to buy? See the tested hardware list with current prices.
So for a single-user local setup:
- llama.cpp — best default. GGUF everywhere, runs on anything, CPU offload.
- vLLM — only worth the setup if you will add concurrent users or agents.
- SGLang — worth it when many requests share a prefix (agents, RAG).
The dramatic numbers you see in benchmarks are concurrency numbers. Do not pick a server for a workload you do not have.
The benchmark everyone quotes
Red Hat’s 2026 comparison of llama.cpp and vLLM is the cleanest public test:
| Concurrent users | Result |
|---|---|
| 1 | Comparable token rate |
| 64 | vLLM delivers roughly 44x more tokens per second |
And latency is worse than throughput. At 64 users, llama.cpp’s P99 time-to-first-token grows exponentially — past three minutes before the first token. vLLM’s P99 TTFT stays nearly flat.
Source: Red Hat Developer: llama.cpp vs vLLM
Why the gap is architectural, not tuning
The 44x gap is not a config problem you can flag your way out of.
- llama.cpp processes requests one at a time through a sequential queue. It is CPU-first by design, with GPU offload added on top. Request 64 waits for requests 1 through 63.
- vLLM uses continuous batching: its scheduler interleaves generation across requests, so new requests join in-flight batches. PagedAttention manages KV cache in blocks so memory fragments less.
Neither design is wrong. They target different jobs. A sequential queue is fine — even efficient — when the queue never has more than one item.
Where SGLang fits
SGLang’s distinct idea is RadixAttention: it stores KV cache in a radix tree and reuses it across requests that share a prefix, managed as an LRU cache.
That matters for one workload shape: agents. An agent pipeline sends the same system prompt and tool definitions on every call. With SGLang, that shared prefix is computed once and reused. Measured results on agent workloads with a shared system prompt: 75–95% cache hit rates on multi-turn conversations (tensorfoundry.io).
If you run OpenClaw-style tool loops against a local server, prefix reuse is a real speedup even at low concurrency — every tool call replays the same long preamble.
Format lock-in: the practical deciding factor
For most home users, the engine choice is made by the model files you already have:
| Engine | Formats | Quantization |
|---|---|---|
| llama.cpp | GGUF | Full GGUF quant ladder; int8 activation quant on CUDA/Vulkan |
| vLLM | safetensors | FP8, INT8, INT4 |
| SGLang | safetensors | Similar HF-checkpoint path |
vLLM’s GGUF support moved to an out-of-tree plugin (vllm-gguf-plugin) that the docs call “highly experimental and under-optimized” (d-central.tech). Do not run your GGUF library through vLLM and conclude vLLM is slow.
Decision table
| Your situation | Better default |
|---|---|
| One user, one chat window | llama.cpp (or Ollama on top of it) |
| Your models are GGUF | llama.cpp |
| CPU offload needed (model bigger than VRAM) | llama.cpp — see MoE offload flags |
| Family/team server, 5+ users | vLLM |
| Agent pipelines with shared system prompts | SGLang |
| RAG with a long fixed context | SGLang |
| You might scale later | Start with llama.cpp; switch when the queue is real |
How to test with your own workload
- Run your real workload, not
--benchmark. One editor session, one agent loop. - Measure time-to-first-token on your longest prompt, not decode tok/s.
- If you serve others: simulate them. 8 parallel
curlloops shows the queue problem in minutes. - For agents, count repeated prefix tokens per call. Above ~2K shared tokens per call, try SGLang.
Final recommendation
Single user: llama.cpp. It ties the others where you live (batch size 1) and beats them on format support and hardware reach.
Adopt vLLM when concurrent users exist, and SGLang when agent or RAG traffic repeats the same prefix all day. The engines are specialists; the benchmark charts only disagree because they measure different jobs.
Sources:
- Red Hat Developer: llama.cpp vs vLLM — choosing the right local LLM inference engine
- tensorfoundry.io: LLM Inference Servers Compared
- d-central.tech on vLLM GGUF plugin status
Related guides
Before you order parts, check the tested hardware list for current prices by tier.
Need OpenClaw fixed live?
Remote rescue sessions for gateway, auth, tunnel, VPS, and model access problems.
See Rescue Session