vLLM Tensor Parallel on Two GPUs: Moving Off Ollama

Bottom Line

• The command is one flag: vllm serve <model> --tensor-parallel-size 2. vLLM cuts every layer across both GPUs.
• Stop Ollama first. vLLM reserves 92% of each GPU’s memory by default (--gpu-memory-utilization 0.92). The two engines cannot share the cards.
• Do not reuse your Ollama GGUF files. vLLM calls GGUF support “highly experimental and under-optimized.” Download an AWQ, GPTQ or FP8 checkpoint.
• Use 2 or 4 GPUs, not 3. The GPU count must divide the model’s attention heads.
• The win is throughput, not single-chat speed. One published dual-3090 test measured 483 output tok/s without NVLink and 715 with it — on a 200-prompt batch.
• Stay on Ollama if you are one person chatting with one model that already fits one card.

Who Should Make This Move

Tensor parallelism is a server feature. It pays when many requests arrive at once: an agent that fans out tool calls, a team sharing one box, or a batch job.

The honest caveat nobody puts first: the official vLLM recipe says an Int4 Qwen3.6 27B fits a single 24GB GPU. If that is your model, a second card under TP buys speed and context room, not the ability to run it. Decide which one you need before you rebuild your stack.

Step 1: Check the Head Count Before Anything Else

Tensor parallelism gives each GPU an equal slice of the attention heads. If the slice is not whole, vLLM stops with:

ValueError: Total number of attention heads (X) must be divisible by tensor parallel size (Y).

Read num_attention_heads in the model’s config.json. Verified values for common models:

All four divide by 2 and 4. Three cards is where builds break, which is one more reason the four-card question is really a two-or-four question.

Step 2: Pick a Checkpoint vLLM Serves Well

This is the part Ollama users trip on. Ollama pulls GGUF files. vLLM works best with Hugging Face checkpoints in safetensors format.

The FP8 row hides a useful fact. The recipe sizes Qwen3.6-27B-FP8 for a single 40GB GPU. No consumer card has 40GB. Two 3090s have 48GB between them, so TP 2 is how a dual-3090 owner runs the FP8 checkpoint at all. On Ampere it runs as weight-only FP8, which saves memory but does not use FP8 math. A 4090 or 5090 gets full W8A8.

Step 3: The Memory Math

vLLM does not load “as much as it needs.” It reserves a fixed share of each card and fills the rest with KV cache.

Worked example, Qwen3.6 27B AWQ Int4 on two 24GB cards. The cyankiwi/Qwen3.6-27B-AWQ-INT4 weights total 20.44 GB across four files.

These are our arithmetic from the file sizes and the default setting, not a measurement. vLLM prints the real KV capacity at startup. Read that line before you trust any number here, including ours.

That spare ~11 GB per card is the real reason to use TP on a model that already fits one card: it becomes context length and concurrent sequences. Our KV cache sizing guide turns it into tokens.

Step 4: The Command

Text-only serving of Qwen3.6 27B FP8 across two cards, adapted from the official vLLM recipe:

vllm serve Qwen/Qwen3.6-27B-FP8 \
  --tensor-parallel-size 2 \
  --max-model-len 32768 \
  --language-model-only \
  --reasoning-parser qwen3 \
  --enable-prefix-caching

What each flag does:

• --tensor-parallel-size 2 — split every layer across both GPUs. Default is 1.
• --max-model-len 32768 — cap the context. If you leave it out, vLLM takes the length from the model config, which is 262,144 for Qwen3.6. That may not fit your KV budget. Start small and raise it.
• --language-model-only — skip the vision encoder. Qwen3.6 is multimodal; text-only saves memory.
• --reasoning-parser qwen3 — the recipe marks this as required for reasoning output.
• --enable-prefix-caching — reuse the shared system prompt across agent turns.

The server speaks the OpenAI API, so most agent tools only need a new base URL.

Step 5: The Two Errors You Will Probably See

1. NCCL P2P failure on a PCIe-only box. vLLM’s troubleshooting page names this one: on some machines without NVLink you may see transport/shm.cc:590 NCCL WARN Cuda failure 217. Its fixes, in order:

export NCCL_CUMEM_HOST_ENABLE=0   # first try, or update the driver
export NCCL_P2P_DISABLE=1          # last resort; the docs warn it may cost speed

To test GPU-to-GPU communication without vLLM in the way, the same page gives NCCL_DEBUG=TRACE torchrun --nproc-per-node=2 test.py.

2. Out of memory at startup. Lower --max-model-len first. Lower --gpu-memory-utilization only if something else must share the card. --enforce-eager turns off CUDA graphs and frees a little memory at a speed cost.

What the Speedup Looks Like

The clearest public data is one benchmark on RTX 3090s: Qwen2.5-7B-Instruct-1M, 200 prompts, 220W per card, with NCCL_P2P_DISABLE toggled to switch NVLink on and off.

Read it with two limits in mind. It is a 7B model and a batch of 200 prompts — a throughput test, not one person waiting on one answer. And four cards did not beat two. For the NVLink half of this story, see Is NVLink worth it for local LLMs?.

vLLM’s own docs add one more rule: when GPUs lack NVLink and the split is uneven, they recommend pipeline parallelism over tensor parallelism. Tensor parallel wants matched cards and a fast link.

Hardware: Matched Cards Matter

TP runs at the pace of the slowest card, so buy a matched pair. The RTX 3090 is still the default pair for this build: 24GB each, and the last GeForce card that takes an NVLink bridge.

EVGA GeForce RTX 3090 24GB — check current listings; used prices moved a lot in 2026. Run each card through our used RTX 3090 checklist before the return window closes.

Two 3090s draw real power under TP, because both cards work at once instead of taking turns. Size the supply with our PSU guide.

See Also

• Is NVLink Worth It for Local LLMs? — the bridge only pays under the tensor-parallel setup on this page
• vLLM vs llama.cpp vs SGLang for a Single User — when one user should not switch engines
• Dual RTX 3090 vs RTX 5090 for Local LLMs — 48GB across two cards vs 32GB on one
• Best 48GB VRAM Setup for Local LLMs — every route to 48GB, including single cards
• Motherboard and CPU for a Multi-GPU LLM Rig — slot lanes and spacing for two cards
• How Much VRAM for 128K Context? — turn the spare memory into context length

Sources

• vLLM docs: parallelism and scaling (--tensor-parallel-size, pipeline-parallel guidance without NVLink); engine arguments (--gpu-memory-utilization default 0.92, --max-model-len, --enforce-eager); troubleshooting (NCCL Cuda failure 217, NCCL_CUMEM_HOST_ENABLE=0, NCCL_P2P_DISABLE=1, torchrun test); GGUF (“highly experimental and under-optimized”, vllm-gguf-plugin); FP8 (W8A8 on Ada/Hopper/Blackwell, W8A16 on Turing/Ampere via Marlin) — read 2026-09-10
• vLLM recipes, Qwen/Qwen3.6-27B — commands, FP8 on a single 40GB GPU, Int4 on a single 24GB GPU, vLLM ≥ 0.17.0
• Hugging Face config.json files: Qwen3.6-27B, Qwen3-32B, Qwen3-30B-A3B, gpt-oss-120b (head counts, 262,144 max positions for Qwen3.6)
• Hugging Face cyankiwi/Qwen3.6-27B-AWQ-INT4 file listing — 20.44 GB of weights
• vLLM GitHub discussion #1041 and issue #4232 — the head-divisibility error text
• Himesh P., “vLLM Performance Benchmarks 4x RTX 3090 (Power Limits, and NVLINK)” — the NVLink on/off table
• NVIDIA PAIR Does Not Pool Your GPU Memory — why two PCs on a LAN are not the same as two cards on one bus