Can a 12GB GPU run Qwen 3.8 27B?

Yes, with the GSQ-RCO 2-bit files. Qwen3.8-27B-GSQ-RCO-IQ2_XS is 8,422,841,472 bytes, or 7.84 GiB, and IQ2_S is 8.62 GiB. The model keeps a growing KV cache on only 16 of its 64 layers, so an f16 cache costs 64 KiB per token: 2.0 GiB at 32K context. IQ2_XS plus 32K context is about 9.84 GiB, inside 12 GiB. The 3.50-bit IQ3_S file is 10.96 GiB and leaves no usable room on a 12GB card.

What are GSQ and RCO in the ISTA-DASLab GGUF files?

They are two methods from the Deep Algorithms and Systems Lab at ISTA. GSQ (Gumbel-Softmax Quantization, arXiv 2604.18556) learns the grid assignments and group scales for each tensor at a given GGUF quantization type. RCO (Riemannian Constrained Optimization, arXiv 2605.00649) then picks which quantization type each tensor gets, under an exact whole-file size budget. The result is one standard GGUF where every tensor can use a different quant type. The IQ2_XS file mixes 12 different quant types across 851 tensors.

What does the -mtp version of the GSQ-RCO file add?

It adds the model's 15-tensor Multi-Token Prediction head (blk.64) for speculative decoding in llama.cpp. The cost is exactly 348,470,208 bytes, or 332 MiB, and it is the same in every file: IQ2_XS, IQ2_S and IQ3_XXS all grow by that identical byte count. The base weights are unchanged, so quality is the same. MTP support merged in llama.cpp PR #22673 on 2026-05-16.

Is the 2-bit Qwen 3.8 27B quant good enough?

It depends on the task. The published card shows GSQ-RCO IQ2_XS at 100.3 percent zero-shot recovery, above the BF16 baseline. The same table shows LiveCodeBench v6 falling from 85.71 to 76.57 and GPQA-Diamond from 89.90 to 84.85. So the headline recovery number comes from the easy zero-shot suite, not from reasoning or coding. IQ2_S at 2.75 bpw matches the base model on AIME25 and scores 82.29 on LiveCodeBench. IQ3_S at 3.50 bpw is the first file that holds coding.

← All guides

Models September 19, 2026

Qwen 3.8 27B GSQ-RCO Quants (2026): 27B on 12GB VRAM

ISTA-DASLab quantized Qwen3.8-27B to as little as 2.50 bits per weight and published the per-tensor allocation for every file. The smallest build is 7.84 GiB and fits a 12GB card. The build the authors call task-lossless is 10.96 GiB and does not, once you add context. Here are the measured sizes, the published benchmark cost of each bit rate, and the card each file actually needs.

Bottom Line

The 2-bit files fit a 12GB card. IQ2_XS is 7.84 GiB and IQ2_S is 8.62 GiB. Add a 2.0 GiB KV cache for 32K context and you are still inside 12 GiB.
The file ISTA-DASLab recommends does not. IQ3_S is 10.96 GiB. Add any real context and a 12GB card is out. It needs 16GB.
Mixed precision buys about one bit. At a matched 8.4 GB file size, the card’s own table puts GSQ-RCO IQ2_XS 10.00 points ahead of Unsloth UD-IQ2_S on AIME25.
The “above BF16” claim is a zero-shot claim. IQ2_XS scores 100.3% recovery on the five zero-shot tasks, and loses 9.14 points on LiveCodeBench v6. Read both columns.
Vision costs another 0.87 GiB. The mmproj projector is BF16 and does not shrink with the quant you pick.

Every size on this page is a byte count from the Hugging Face API, read 2026-09-19. Every benchmark number is from the repo’s own model card. We did not run these benchmarks. Nobody has published independent tokens-per-second figures for these specific files.

What ISTA-DASLab Actually Shipped

The repo is ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF. It was created 2026-08-28 and last modified 2026-09-02. The API reports 1,154,265 downloads and 1,432 likes.

Two methods made the files:

GSQ (Gumbel-Softmax Quantization, arXiv 2604.18556) quantizes each tensor at a given GGUF type. It learns the per-coordinate grid assignments and the per-group scales together.
RCO (Riemannian Constrained Optimization, arXiv 2605.00649) then chooses which type each tensor gets. It treats the size budget as a smooth manifold in logit space, so the budget is met exactly.

The output is a plain GGUF. It runs in stock llama.cpp, Ollama and LM Studio. No fork, no patch.

File Sizes, in GiB

The card lists sizes in decimal GB. Your GPU reports GiB. Here is the conversion, from the raw byte counts in the API.

File	bpw	Bytes	GiB
`GSQ-RCO-IQ2_XS`	2.50	8,422,841,472	7.84
`GSQ-RCO-IQ2_S`	2.75	9,259,510,912	8.62
`GSQ-RCO-IQ3_XXS`	3.00	10,094,357,632	9.40
`GSQ-RCO-IQ3_S`	3.50	11,771,546,784	10.96
`mmproj-Qwen3.8-27B-BF16`	16	931,146,528	0.87

The -mtp build of each file is larger. See the next section for the exact amount.

Against Unsloth’s uniform quants of the same base model, read from the same API:

Unsloth file	GiB	Nearest GSQ-RCO file	GiB
`UD-IQ2_S`	7.80	`GSQ-RCO-IQ2_XS`	7.84
`UD-Q2_K_XL`	9.15	`GSQ-RCO-IQ3_XXS`	9.40
`UD-IQ3_S`	11.21	`GSQ-RCO-IQ3_S`	10.96
`UD-IQ4_XS`	13.27	—	—

GSQ-RCO IQ3_S is 0.25 GiB smaller than Unsloth’s UD-IQ3_S. It is not a smaller class of file. It is the same class, packed differently.

The Thing Nobody Else Says About the -mtp Files

The card says the -mtp builds are “about 0.35 GB larger.” The byte counts say something sharper.

Pair	Delta in bytes
IQ2_XS → IQ2_XS-mtp	348,470,208
IQ2_S → IQ2_S-mtp	348,470,208
IQ3_XXS → IQ3_XXS-mtp	348,470,208
IQ3_S → IQ3_S-mtp	348,470,176

That is the same 332 MiB, to the byte, in three of four pairs, and within 32 bytes in the fourth. Our arithmetic, from the API byte counts.

The reason is in the repo’s own allocation dumps. We read tensor-allocation/Qwen3.8-27B-GSQ-RCO-IQ2_XS-mtp.rco-allocation.txt. The MTP head is block blk.64, 15 tensors. Every one of its 8 weight tensors is Q6_K. The other 7 are F32 norms.

Now compare that to the model the head drafts for. In the same IQ2_XS file, the quant-type histogram is BF16=96, F32=360, IQ1_M=31, IQ1_S=36, IQ2_S=60, IQ2_XS=53, IQ2_XXS=70, IQ3_S=47, IQ3_XXS=37, IQ4_XS=11, Q2_K=53, Q4_K=4, Q6_K=8. The only 8 Q6_K tensors in the entire 866-tensor file are the MTP head. The token embedding is IQ1_M. The output layer is IQ4_XS.

In the 2.50-bit build, the speculative draft head is the highest-precision weight block in the file — higher than the model it drafts for. RCO does not spend its budget on it, because the head sits outside the target bit-width. The allocation dump states the whole-file bpw rises from 2.5000 to 2.5652 once the head is counted.

Two practical consequences:

The MTP tax is flat, not proportional. It is 4.1% on top of IQ2_XS and 3.0% on top of IQ3_S. If you are 300 MiB from fitting a card, skip the -mtp file.
Speculative decoding here does not degrade with the base quant. The drafter stays at Q6_K whether you run 2.50 bits or 3.50 bits.

MTP support in llama.cpp merged in PR #22673 on 2026-05-16. Nobody has published an accepted-token rate for these specific GSQ-RCO heads. The repo ships a speculative-decoding plot, but no numeric table, so we will not quote a speedup.

What the Low Bit Rate Costs You

All of this is the repo’s own results table. We reproduce it as published.

Variant	bpw	ZS avg	recovery	AIME25	GPQA-D	LCB v6
BF16	16.00	74.34	100.0%	100.00	89.90	85.71
GSQ-RCO IQ2_XS	2.50	74.54	100.3%	96.67	84.85	76.57
GSQ-RCO IQ2_S	2.75	75.70	101.8%	100.00	86.36	82.29
GSQ-RCO IQ3_XXS	3.00	74.81	100.6%	100.00	88.89	84.57
GSQ-RCO IQ3_S	3.50	74.47	100.2%	100.00	89.39	85.71
UD-IQ2_S	2.49	73.80	99.3%	86.67	76.26	72.00
UD-Q2_K_XL	2.88	74.37	100.0%	100.00	86.87	82.28
UD-IQ3_S	3.52	75.49	101.5%	96.67	89.90	84.00

Read the LiveCodeBench column, not the recovery column. Recovery is the zero-shot average, and it is above 100% for every GSQ-RCO file. That sounds like free compression. It is not.

2.50 bpw costs 9.14 points on LiveCodeBench and 5.05 on GPQA-Diamond.
2.75 bpw recovers most of it. It ties BF16 on AIME25 and reaches 82.29 on LiveCodeBench.
3.50 bpw ties BF16 exactly on AIME25 and LiveCodeBench, and trails by 0.51 on GPQA-Diamond.

The mixed-precision gain is clearest at matched size. At 8.4 GB, GSQ-RCO IQ2_XS beats Unsloth UD-IQ2_S by 10.00 points on AIME25, 8.59 on GPQA-Diamond and 4.57 on LiveCodeBench v6. That is the whole argument for these files.

KV Cache Math

Qwen3.8-27B uses hybrid attention. Its config.json lists 64 layers and full_attention_interval: 4, so only 16 layers keep a growing cache. The other 48 are linear-attention layers with a fixed state.

2 (K and V) x 16 layers x 4 KV heads x 256 head dim x 2 bytes = 65,536 bytes per token

That is 64 KiB per token in f16. This is our arithmetic from the published config, not a measured figure.

Context	f16 KV cache
8K	0.50 GiB
32K	2.00 GiB
64K	4.00 GiB
128K	8.00 GiB
262K (native max)	16.00 GiB

These sums exclude compute buffers and the linear-attention state, which we did not measure. Treat each row as a lower bound.

Fit by VRAM Tier

Weights plus f16 KV cache. Vision adds 0.87 GiB on top of any row.

File	12 GiB card	16 GiB card	24 GiB card
IQ2_XS (7.84)	32K fits (9.84); 64K tight (11.84)	128K fits (15.84)	262K tight (23.84)
IQ2_S (8.62)	32K fits (10.62); 64K no (12.62)	64K fits (12.62); 128K no (16.62)	128K fits (16.62)
IQ3_XXS (9.40)	32K very tight (11.40)	64K fits (13.40)	128K fits (17.40)
IQ3_S (10.96)	No usable context	32K fits (12.96)	128K fits (18.96)
IQ3_S-mtp (11.29)	No	32K fits (13.29)	128K fits (19.29)

The headline “27B on a 12GB card” is true, and it is true only at 2 to 2.75 bits. The file the authors call task-lossless needs 16GB before you load a single token of context. If you want IQ3_S with vision and 32K context, that is 10.96 + 0.87 + 2.00 = 13.83 GiB, and a 12GB card cannot hold it.

Which GPU to Buy

Prices are US street ranges as of August 2026, from our hardware price reference. The 2026 DRAM shortage moved these hard. Check current listings before you buy.

If you only want the model to run: a 12GB card. The Intel Arc B580 12GB is the cheapest new 12GB card at $300–310 (as of August 2026). It runs IQ2_S at 32K context. Accept the LiveCodeBench drop, or use IQ2_S rather than IQ2_XS to keep most of it.

Check Arc B580 12GB prices on Amazon

If you want IQ3_S, which is the point of this release: 16GB. The RTX 5060 Ti 16GB runs the task-lossless file at 32K context, or the -mtp build with vision at 8K. It sits at $589–805 (as of August 2026), well above its $429 MSRP. This is the smallest card that holds the quality the benchmark table promises.

Check RTX 5060 Ti 16GB prices on Amazon

If you want IQ3_S at 128K context: 24GB. IQ3_S plus an 8.00 GiB cache is 18.96 GiB. A used RTX 3090 holds it with headroom for the projector and compute buffers. Used listings ran $1,000–1,300 (as of August 2026), and one scraped tracker showed $1,399 on 2026-09-16. That range is drifting up.

Check RTX 3090 24GB prices on Amazon

Do not buy a larger card for this model alone. At 3.50 bits the file stops improving. Spend the difference on a second model or on RAM.

How to Run It

Pull one file, not the repo. The files are single-part.

hf download ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF \
  Qwen3.8-27B-GSQ-RCO-IQ3_XXS.gguf --local-dir .

llama-server -m Qwen3.8-27B-GSQ-RCO-IQ3_XXS.gguf -ngl 99 -fa on -c 32768

For vision, add the projector:

hf download ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF \
  mmproj-Qwen3.8-27B-BF16.gguf --local-dir .

llama-mtmd-cli -m Qwen3.8-27B-GSQ-RCO-IQ3_XXS.gguf \
  --mmproj mmproj-Qwen3.8-27B-BF16.gguf --image photo.jpg -p "Describe this image."

Audit the build before you trust it. The repo ships tensor-allocation/<file>.rco-allocation.txt for every GGUF. It lists the quant type of every tensor plus the histogram. Almost no quant repo publishes this. Read it before you argue about quality.

Sources

Hugging Face API, ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF?blobs=true (downloads, likes, createdAt 2026-08-28, lastModified 2026-09-02, all file byte counts), read 2026-09-19
ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF README.md (bpw table, results table, usage), read 2026-09-19
tensor-allocation/Qwen3.8-27B-GSQ-RCO-IQ2_XS.rco-allocation.txt and the -mtp, IQ2_S, IQ3_XXS, IQ3_S dumps (quant-type histograms, blk.64 head), read 2026-09-19
arXiv 2604.18556 (GSQ) and arXiv 2605.00649 (RCO), as cited by the model card
Hugging Face API, unsloth/Qwen3.8-27B-GGUF?blobs=true (baseline file sizes), read 2026-09-19
Qwen/Qwen3.8-27B config.json (64 layers, full_attention_interval: 4, 4 KV heads, head dim 256, 262,144 max positions), read 2026-09-19
GitHub API, ggml-org/llama.cpp PR #22673 “llama + spec: MTP Support”, merged 2026-05-16, read 2026-09-19
OpenClaw DC hardware price reference, compiled 2026-08-11, last updated 2026-09-16