Bonsai 2 27B on RTX 3060 12GB: Fits, Needs a Fork (2026)

Bottom Line

• Yes, a 27B model fits a 12GB RTX 3060. Ternary Bonsai 2 27B is 7.21 GB (PQ2_0) or 5.95 GB (PTQ1_0). Both fit fully on the GPU. PTQ1_0 also fits an 8GB card, with short context.
• You need Prism ML’s llama.cpp fork. Stock llama.cpp, Ollama and LM Studio do not run these files as of 2026-09-17.
• The quality numbers are vendor claims. Prism ML reports 84.78 against 86.32 for FP16 (98.2%). No one else has checked it yet. The model was published on 2026-09-16.
• No one has measured a 3060 yet. Our estimate is about 25 to 35 tok/s. The method to measure your own is below.
• If you already own a 24GB card, run normal Qwen 3.8 27B instead. By the vendor’s own table, the 4-bit file scores higher (85.18) and runs on stock tools.

What Bonsai 2 27B Is

Item	Value
Maker	Prism ML (prism-ml on Hugging Face)
Base model	Qwen/Qwen3.8-27B
License	Apache 2.0
Parameters	27.36B (24.35B backbone, 2.54B embedding and LM head, 0.46B vision)
Average bits per weight	1.72 (ternary weights: -1, 0, +1)
Native context	262K tokens
Published	2026-09-16
Backends	Fork of llama.cpp (CUDA, Metal, CPU), MLX fork for Apple

File sizes from the Hugging Face API tree, read 2026-09-17:

File	Bytes	GiB	Fits 12GB?	Fits 8GB?
PTQ1_0 (dense trits)	5,946,648,928	5.54	Yes	Yes, short context
PQ2_0 (2-bit slots)	7,206,168,928	6.71	Yes	Too tight
F16 reference	53,808,408,928	50.11	No	No
mmproj Q8_0 (vision)	629,246,976	0.59	Optional	Optional

For comparison, the normal 4-bit Qwen3.8-27B file (Unsloth UD-Q4_K_XL) is 16.35 GiB. That file needs a 24GB card.

The Fork Requirement

The model card is direct: “Stock llama.cpp will not run these files.” It rejects PQ2_0 and PTQ1_0 as unknown types. It also loads the older Q2_0 type with no warning and gives garbage output.

What this means for you:

• Ollama: no. It uses upstream llama.cpp.
• LM Studio: no. Same reason.
• Upstream llama.cpp: not yet. Issue #29058, “Support Prism PQ2_0 (GGML type 142) and PTQ1_0 (type 143)”, is open. We found no merged pull request for these types.
• The fork: yes. PrismML-Eng/llama.cpp is a GitHub fork of ggml-org/llama.cpp. Its latest release on 2026-09-17 was prism-b10687-5d80cff.

The card lists CUDA, Metal and CPU. It does not list Vulkan or SYCL. So we do not recommend an Intel Arc or AMD card for this model yet.

A fork is a real cost. You update it by hand, and upstream fixes reach it late. Prism ML says its Bonsai-demo repo is the source of truth for tested setups.

Quality: Vendor Numbers Only

Prism ML ran 14 benchmarks in thinking mode with EvalScope and vLLM on an H100. We did not re-run them.

Build of Qwen3.8-27B	Size	Average (14)	vs FP16
FP16	54 GB	86.32	100%
Unsloth UD-Q4_K_XL	17.6 GB	85.18	98.7%
Bonsai 2 27B	5.80 GB	84.78	98.2%
IQ2_XXS	9.4 GB	72.59	84.1%

The line that matters: in the vendor’s own table, the normal 4-bit file still scores higher than Bonsai 2. Bonsai’s win is size, not quality. It beats the only conventional file of similar size (IQ2_XXS) by 12 points.

Treat this as a vendor claim until a third party repeats it. The earlier Ternary Bonsai 27B release drew upstream bug reports from users who could not get it to run (issues #25727 and #26073 on ggml-org/llama.cpp, both now closed).

KV Cache Headroom on 12GB (Our Estimate)

Qwen3.8-27B uses hybrid attention. Its config.json lists 64 layers. Only every fourth layer is full attention, so 16 layers keep a cache that grows. The other 48 are linear-attention layers with a fixed-size state.

Growing cache per token, f16:

2 (K and V) x 16 layers x 4 KV heads x 256 head dim x 2 bytes = 65,536 bytes = 64 KiB per token

Fixed linear-attention state (config says mamba_ssm_dtype: float32):

48 layers x 48 value heads x 128 key dim x 128 value dim x 4 bytes = 151 MB (about 0.14 GiB)

This assumes the Bonsai GGUF keeps the base model’s attention layout. The card says it does (“~75% linear attention”).

Context	f16 KV	+ PQ2_0 (6.71) + state (0.14) + 1 GiB buffers	Fits 12 GiB?
32K	2.0 GiB	9.85 GiB	Yes
64K	4.0 GiB	11.85 GiB	Just, with no desktop on the card
128K	8.0 GiB	15.85 GiB	No (try a q8_0 cache: about 11.85 GiB)

The 1 GiB for compute buffers and CUDA context is our assumption, not a measurement. If your monitor runs on the same card, subtract what nvidia-smi shows in use before you load.

On an 8GB card: PTQ1_0 (5.54) + state (0.14) + 1 GiB buffers = 6.68 GiB. That leaves about 1.3 GiB, so plan on about 16K context in f16.

Whether the fork supports a quantized KV cache (-ctk q8_0 -ctv q8_0) with these kernels is not documented on the card. Test it.

Speed: Vendor Numbers and Our Estimate

Prism ML measured decode speed with llama-bench (tg128, batch 1, no vision tower). No RTX 3060 or other 12GB card is in its table.

Card	PQ2_0 tok/s	PTQ1_0 tok/s
RTX 5090	129.9	120.5
RTX 4090	81.2	91.1
NVIDIA L4 (72 W)	29.8	32.1
Apple M5 Pro laptop	28.1	—

Our 3060 estimate, and how we got it. Decode speed on a batch-1 run is limited by memory bandwidth. The 3060 has 360 GB/s. Reading the 7.21 GB file once per token gives a hard ceiling of 360 / 7.21 = about 50 tok/s. Real runs reach a fraction of the ceiling. The vendor’s L4 row is the nearest measured card of the same class. So we expect about 25 to 35 tok/s. This is an estimate, not a measurement.

The vendor says PTQ1_0 wins on older and memory-bound cards, and PQ2_0 wins on newer ones. The 3060 is an Ampere card, so test both files.

Measure Your Own tok/s

1. Get the fork. Use a prebuilt archive from the releases page, or build it with CUDA:

git clone https://github.com/PrismML-Eng/llama.cpp && cd llama.cpp
cmake -B build -DGGML_CUDA=ON && cmake --build build -j

2. Download one file:

hf download prism-ml/Ternary-Bonsai-2-27B-gguf Ternary-Bonsai-2-27B-PQ2_0.gguf --local-dir .

3. Run the same test the vendor ran, so your number compares to theirs:

./build/bin/llama-bench -m Ternary-Bonsai-2-27B-PQ2_0.gguf -ngl 99 -fa 1 -p 512 -n 128

4. Repeat with the PTQ1_0 file. Keep the faster one.
5. Repeat at the context you really use. Add -d 32768 to llama-bench to test at a filled cache, if your build has the flag.
6. Watch nvidia-smi during the run. If memory is near 12 GiB, lower -c in real use.

This is a reasoning model and it thinks by default. The vendor recommends --temp 1.0 --top-p 0.95 --top-k 20. Thinking tokens count against your speed, so judge it on time to a finished answer. See usable tokens per second by task.

Bonsai on a 3060 vs Qwen 3.8 27B on a 3090

	Bonsai 2 27B on RTX 3060 12GB	Qwen3.8-27B UD-Q4_K_XL on RTX 3090 24GB
Model file	7.21 GB	16.35 GiB
Quality (vendor table, 14 benchmarks)	84.78	85.18
Speed	not measured; about 25-35 tok/s estimated	about 40 tok/s community-measured, 66 with MTP
Context in VRAM	about 64K (estimate)	about 64K at f16, 128K with IQ4_XS
Software	PrismML fork only	Stock llama.cpp, Ollama, LM Studio
Card price, as of September 2026	$329-460 new, about $260 used	about $1,000-1,300 used (one tracker showed $1,399)

The 3090 path is the safe one: better scores, measured speed, standard tools. The 3060 path costs about a third as much and depends on one vendor’s fork and one vendor’s numbers. Details on the 3090 side: Qwen 3.8 27B on RTX 3090.

What to Buy

If you have no GPU and a budget near $400, the RTX 3060 12GB is now a real 27B card. Before this model, 12GB meant 8B to 14B models. Prices are as of September 2026 and move often: $329 to $460 new, about $260 used.

Check RTX 3060 12GB prices on Amazon

If you want room to grow, the RTX 5060 Ti 16GB gives 4GB more for context or a second model. It costs $589 to $805 as of September 2026. It is a CUDA card, the backend the fork lists. No vendor speed number exists for it, and we did not test it.

Check RTX 5060 Ti 16GB prices on Amazon

If you already own a 12GB card, do not buy anything. Download the file and run the benchmark above.

Sources

• Hugging Face model card, prism-ml/Ternary-Bonsai-2-27B-gguf (license, parameters, 1.72 bpw, fork requirement, benchmark and throughput tables), read 2026-09-17
• Hugging Face API file tree and model API for the same repo (file sizes in bytes, created 2026-09-16T23:40 UTC), read 2026-09-17
• Qwen/Qwen3.8-27B config.json (64 layers, full attention every 4th layer, 4 KV heads, head dim 256, 48 linear value heads of 128x128, float32 SSM state), read 2026-09-17
• GitHub API, PrismML-Eng/llama.cpp (fork of ggml-org/llama.cpp, latest release prism-b10687-5d80cff), read 2026-09-17
• GitHub issue search, ggml-org/llama.cpp: issue #29058 (open), #25727 and #26073 (earlier Bonsai 27B bug reports), read 2026-09-17
• RTX 3060 12GB specifications (192-bit, 15 Gbps GDDR6, 360 GB/s): GIGABYTE, MSI and ASUS spec pages
• Price reference: OpenClaw DC hardware price file, updated 2026-09-16 (RTX 3060, RTX 5060 Ti, used RTX 3090 rows)

See Also

• Best local LLM for the RTX 3060 12GB
• Qwen 3.8 27B on RTX 3090
• Lowest KV cache models for long context
• Ollama vs llama.cpp
• llama.cpp flags explained
• Best GPU under $500 for local AI