Can I Run Underdog Saluki 27B on 8GB VRAM? (2026): Not Fully

Bottom Line

• The file is under 8 GB. Your 8GB GPU is not big enough for it plus context. The GGUF is 7,898,369,152 bytes: 7.89 GB, or 7,532 MiB. An 8GB card has 8,192 MiB.
• Full GPU offload leaves you 269 MiB at best, and 54 MiB short at worst. That is before the KV cache and the driver take their share. In practice, that is zero usable context.
• Partial offload works. Put 13 layers on the CPU (-ngl 52) for 16K of q8_0 context. Put 17 there (-ngl 48) for 32K. The speed ceiling drops by 25-40%.
• The card’s own -c 32768 -ngl 99 command needs about 10.2-10.5 GiB. It fits a 12GB card, not an 8GB one.
• The fact most pages miss: llama.cpp’s default --fit keeps 1,024 MiB free on each GPU. On an 8GB card that leaves 7,168 MiB, and the GPU part of the weights alone is 7,257 MiB. So stock llama.cpp will move layers to the CPU by itself, unless you force -ngl 99.
• If you are buying a card for this model, buy 12GB. An Arc B580 runs it with 32K of context on the GPU.

What the Model Is

All figures below come from the Underdog Saluki model card and the Hugging Face API, read on 14 October 2026.

Spec	Value
Repo	ConwayResearch/Underdog-Saluki-27B-1.0
Base model	Qwen/Qwen3.8-27B, via ISTA-DASLab Qwen3.8-27B-GSQ-RCO-GGUF
File	Underdog-Saluki-27B-1.0-IQ2-mix.gguf, 7,898,369,152 bytes
Vision add-on	F16 927,606,912 bytes, or Q8_0 629,246,976 bytes
Runtime	Stock llama.cpp, and apps built on it
License	Apache 2.0
Created on Hugging Face	8 October 2026
Thinking	On by default, switchable per request

We also read the GGUF header itself. The file holds 851 tensors. Its metadata names the quantizer as “ISTA DASLab” and the model as “Qwen3.8-27B GSQ-RCO”. So Saluki is a new mix of the GSQ-RCO tensors at a smaller size budget, not a new training run.

Two tensors matter for VRAM:

Tensor	Quant	Size	Where llama.cpp puts it
token_embd.weight	IQ1_M	265.2 MiB	Always CPU RAM
output.weight	IQ3_S	521.0 MiB	GPU, with the last layer
64 transformer blocks	mixed, 2-4 bit	80 to 155 MiB each	GPU, last layers first

The llama.cpp source says it directly: “there is very little benefit to offloading the input layer, so always keep it on the CPU.” That is why the GPU needs 7,257 MiB of weights, not 7,532.

What the card claims about quality

These are Underdog’s numbers. We did not run them.

Benchmark	Saluki	Full-size Qwen3.8-27B
Tool calling, Underdog Bench (of 120)	88	84
Parallel tool calls (of 100)	42	35
SWE-bench Verified (of 50)	30	33
MuSR	67.5	79.6 (public)
AIME 2025 (avg@4)	79.2	96.7 (public)

The card says the “public” scores come from a different harness. So the MuSR and AIME gaps are not a like-for-like test. The tool-calling rows are like-for-like: Underdog ran both models in the same harness. The card also lists Bonsai 2 at 5.95 GB and 70 of 120 on tool calling.

Why “Under 8 GB” Does Not Mean “Fits 8GB”

Three numbers decide it: the weights, the fixed state, and the compute buffer. The KV cache comes after them.

1. The growing KV cache. Qwen3.8-27B’s config.json lists 64 layers with full_attention_interval: 4. Only 16 layers keep a cache that grows with context.

f16:  2 (K and V) x 16 layers x 4 KV heads x 256 head dim x 2 bytes = 65,536 bytes = 64 KiB per token
q8_0: same 32,768 values x 1.0625 bytes                                = 34,816 bytes = 34 KiB per token

2. The fixed linear-attention state. The other 48 layers hold a state that does not grow. llama.cpp stores it in F32.

recurrent state: 48 value heads x 128 x 128               = 786,432 floats per layer
conv state:      3 x (6,144 + 2 x 16 x 128)               =  30,720 floats per layer
48 layers x 817,152 floats x 4 bytes                      = 149.6 MiB per sequence

That is per sequence. llama-server now defaults to 4 slots (“n_parallel = 4 and kv_unified = true” in the server source). llama.cpp sizes this state by slot count, so the default costs 598 MiB. Add -np 1 on a small card and it costs 150 MiB.

3. The compute buffer. We did not measure it. The nearest public figure is llama.cpp issue #29063. There, Qwen3.8-27B reports 516 MiB per GPU on build b10680 and 839 MiB on later builds, at the default batch size. That was a two-GPU SYCL setup. We use 516-839 MiB as the range.

The sum on an 8GB card, all layers on the GPU, -np 1:

Item	MiB
GPU card total	8,192
Weights on the GPU	-7,257
Linear-attention state	-150
Compute buffer	-516 to -839
Left for KV cache and driver	+269 to -54

At best, 269 MiB holds about 4,300 tokens of f16 cache. But the CUDA context and any monitor on the card also need memory. So the honest answer is no usable context at -ngl 99.

Fit Table by Card

Weights plus state plus a 839 MiB compute buffer, -np 1, with llama.cpp’s default 1,024 MiB margin left free on GPUs. Mac rows use the Metal working-set limit instead. Every row is our estimate, not a measurement.

Hardware	Memory the model can use	Fits on the GPU?	Context that fits
8GB GPU (RTX 4060, RTX 5060)	8,192 MiB	No, not with context	None at -ngl 99. 16K q8_0 at -ngl 52; 32K q8_0 at -ngl 48
12GB GPU (Arc B580, RTX 3060)	12,288 MiB	Yes	32K f16, or 64K q8_0 (about 10.2 GiB total)
16GB GPU (RTX 5060 Ti 16GB)	16,384 MiB	Yes	About 96K f16, or 128K q8_0 (about 12.3 GiB total)
24GB GPU (RTX 3090, RTX 4090)	24,576 MiB	Yes	128K f16, or the full 262K at q8_0 (about 16.6 GiB total)
16GB Mac	about 12,124 MiB Metal limit	Yes	32K f16 (about 10.3 GiB total)
24GB Mac	about 18 GiB Metal limit (estimate)	Yes	128K f16 (about 16.3 GiB total)

Notes on the Mac rows. A llama.cpp log from a 16GB M2 Mac mini (issue #20704) reports recommendedMaxWorkingSetSize = 12713.12 MB. That is 12,124 MiB, or 74% of 16 GiB. Developers on the Apple developer forum describe the same limit as about 75% of physical memory. We found no 24GB log, so the 24GB row assumes the same 75%. On a Mac the whole 7,532 MiB file counts, because memory is shared.

Vision adds 600 MiB (Q8_0) or 885 MiB (F16) to any row. On an 8GB GPU, skip it.

With llama-server’s default 4 slots, add 449 MiB to every row. On 12GB and up that only trims context. On 8GB it decides the result.

How to Run It on an 8GB GPU

The card’s command forces every layer onto the GPU and asks for 32K:

llama-server -m Underdog-Saluki-27B-1.0-IQ2-mix.gguf --jinja -ngl 99 -fa on -c 32768

On 8GB, that fails or spills. Use this instead for 16K of context:

llama-server -m Underdog-Saluki-27B-1.0-IQ2-mix.gguf --jinja -fa on \
  -ngl 52 -c 16384 -np 1 -ctk q8_0 -ctv q8_0

What each change does:

• -ngl 52 keeps layers 0 to 12 on the CPU. That is 1,201 MiB of weights. llama.cpp offloads the last layers first, so the first layers are the ones that stay behind.
• -np 1 keeps one slot. That saves 449 MiB of linear-attention state.
• -ctk q8_0 -ctv q8_0 cuts the cache from 64 KiB to 34 KiB per token. 16K then costs 544 MiB.

Our sum for this command: 7,589 MiB used on the GPU, with the 839 MiB compute buffer. That leaves about 600 MiB for the driver and the desktop. For 32K, use -ngl 48 -c 32768. That moves 1,537 MiB to the CPU and leaves about 400 MiB.

The easy route: omit -ngl and let --fit choose. It is on by default and only changes settings you did not set. It keeps 1,024 MiB free, which is cautious on 8GB. Lower it with --fit-target 512 if your card drives no monitor.

To check a setting before you commit, run llama-server once and read the log lines that report the model, KV and compute buffer sizes. Then watch nvidia-smi during a long prompt.

Speed: No Measured Numbers Yet

We found no published tok/s for Saluki, or for its GSQ-RCO parent on an 8GB card, as of 14 October 2026. The figures below are ceilings from bandwidth arithmetic, not predictions. Decode at batch 1 reads every GPU weight once per token.

Setup	Arithmetic	Ceiling
RTX 4060, all on GPU (no context)	272 GB/s / 7.61 GB	about 36 tok/s
RTX 4060, -ngl 52, DDR4-3200 dual channel	6.35 GB at 272 GB/s + 1.26 GB at 51.2 GB/s	about 21 tok/s
RTX 4060, -ngl 52, DDR5-5600 dual channel	6.35 GB at 272 GB/s + 1.26 GB at 89.6 GB/s	about 27 tok/s
Arc B580 12GB, all on GPU	456 GB/s / 7.61 GB	about 60 tok/s

The RTX 4060’s 272 GB/s comes from Guru3D’s review (128-bit, 17 Gbps). The B580’s 456 GB/s comes from Intel’s spec page. System RAM figures are channel arithmetic: 2 channels x 8 bytes x transfer rate.

Thinking is on by default, and the card warns that the model “often reasons at length”. For tool calls, the card’s own setting is thinking off and temperature 0. That shortens every reply more than any flag above.

Saluki vs the Other 27B Files Near 8 GB

All sizes from the Hugging Face API, read on 14 October 2026.

File	Bytes	MiB	Runs on	8GB GPU, all layers?
Ternary Bonsai 2 27B PTQ1_0	5,946,648,928	5,671	PrismML llama.cpp fork only	Yes, short context
Ternary Bonsai 2 27B PQ2_0	7,206,168,928	6,872	PrismML fork only	Too tight
Underdog Saluki 27B IQ2-mix	7,898,369,152	7,532	Stock llama.cpp	Not with context
GSQ-RCO IQ2_XS	8,422,841,472	8,033	Stock llama.cpp	No
GSQ-RCO IQ2_S	9,259,510,912	8,831	Stock llama.cpp	No
GSQ-RCO IQ3_XXS	10,094,357,632	9,627	Stock llama.cpp	No

Of these files, Saluki is the only one under 8,192 MiB that runs on stock llama.cpp. It is 500 MiB smaller than GSQ-RCO IQ2_XS. That 500 MiB is the difference between a partial offload and no fit at all.

If you have 12GB, compare it with IQ2_S and IQ3_XXS. Those spend 1.3 to 2.1 GiB more on weights. Our GSQ-RCO quants page has ISTA-DASLab’s own benchmark table for each size. Do not compare those scores directly with the Saluki card. The two cards use different harnesses.

What to Buy

If you own an 8GB card, try it before you buy anything. Use the -ngl 52 command above. Tool calling is the reason to pick this file, and short tool calls do not need 32K.

If you are buying for this model, buy 12GB. It is the smallest tier that holds every layer plus 32K of cache. Prices are as of October 2026 and move weekly.

Card	Price (as of October 2026)	Why pick it
Intel Arc B580 12GB	$329.99 Newegg; $369.99 Amazon	Cheapest new 12GB card; 456 GB/s
RTX 3060 12GB	from $419 new; about $307 used	CUDA, the most tested llama.cpp backend
RTX 5060 Ti 16GB	$589-805 range; $789 at B&H on 24 September	128K of q8_0 context, or room for vision

The Arc B580 12GB is the cheapest way to run Saluki fully on a GPU. If you want CUDA, the RTX 3060 12GB costs about $90 more new. If you also want vision or long documents, the RTX 5060 Ti 16GB holds 128K with the projector loaded.

On a Mac, 16GB is enough. A Mac mini M6 with 24GB costs $1,099 (Apple configurator, read 22 September 2026). That buys the 128K row in the table above.

Honest Caveats

1. We did not run the model. Every fit row is arithmetic from the GGUF header, the Qwen config and the llama.cpp source.
2. The compute buffer is borrowed. The 516-839 MiB range comes from one public report on a two-GPU SYCL setup. CUDA on one card may differ. Lower -ub to shrink it.
3. The -ngl advice leaves 400-600 MiB for the CUDA context and driver. We did not measure that need. A monitor on the same card takes more. Check nvidia-smi before you load.
4. The quality claims are Underdog’s. 120 tasks is a small test, and the card says so.
5. We did not check Ollama or LM Studio. The card names stock llama.cpp. Apps built on it should load the file, but we did not test them.

Sources

• Underdog Saluki 27B 1.0 model card: size claim, quickstart, benchmarks, limitations, license
• Hugging Face API: Saluki file tree: byte sizes; model API for the 8 October 2026 creation date
• Saluki GGUF header, first 25 MB read by HTTP range request on 14 October 2026: 851 tensors, per-tensor quant types and sizes
• Qwen/Qwen3.8-27B config.json: 64 layers, full attention every 4th layer, 4 KV heads, head dim 256, 48 linear value heads, F32 SSM state
• Hugging Face API: ISTA-DASLab GSQ-RCO file tree and prism-ml Ternary Bonsai 2 27B file tree: comparison sizes
• llama.cpp source on GitHub, master branch, read 14 October 2026: src/llama-model.cpp (input layer on CPU, layer placement, F32 recurrent state sized by slot count), tools/server/server.cpp (4 slots by default), common/common.h (--fit on, 1,024 MiB target)
• llama.cpp issue #29063: Qwen3.8-27B compute buffer sizes
• llama.cpp issue #20704: Metal working set on a 16GB M2 Mac mini
• Apple developer forum thread 732035: the ~75% Metal limit
• Guru3D RTX 4060 review and Intel Arc B580 specifications: memory bandwidth
• OpenClaw DC hardware price reference, last updated 13 October 2026: Arc B580, RTX 3060, RTX 5060 Ti and Mac mini rows

See Also

• Best local LLM for 8GB VRAM: what else fits the tier, without offload
• Best local LLM for 12GB VRAM: Qwen3.8 27B at 3 bits, the step up from Saluki
• Qwen 3.8 27B GSQ-RCO quants: the parent files, their benchmarks and the MTP build
• Bonsai 2 27B on RTX 3060 12GB: the smaller 27B file that needs a fork
• Best local LLM for a 16GB MacBook Pro: how to raise the Metal limit
• KV cache quantization: q8 vs q4: what -ctk q8_0 costs in quality
• llama.cpp flags explained: -ngl, -np, -ub and --fit
• Can I run OpenClaw on 8GB RAM and 8GB VRAM?: the agent side of the same tier