Dense 70B or MoE 120B: Which Is the Better Local Model in 2026?

Bottom Line

• Same footprint, different shape. Llama 3.3 70B at Q4_K_M is 42.5GB. Laguna S 2.1 at UD-Q2_K_XL is 39.7GB. On disk they are the same tier.
• The 70B costs 327,680 bytes of KV cache per token. Laguna costs 49,152. 80 full-attention layers against 12.
• On 48GB: under 20K tokens of context for the dense 70B, 100K+ for the MoE. Same card.
• The MoE reads ~8x fewer bytes per token, so its bandwidth ceiling is several times higher. Derived, not benchmarked.
• The MoE gives up prompt caching on its sliding-window layers. That is the tradeoff nobody prints.
• Pick the dense 70B only if you need that specific model at short context. Otherwise the MoE is the better local citizen at 48GB and at 96GB.

The Two Models, From Their Config Files

The 24GB version of this argument compares a 35B MoE with a dense 27B. This page moves it up to the tier where the choice is forced: a dense 70B against the 118B and 117B mixtures that now occupy the same disk space. All figures below are read from the published config.json of each model.

One rule carries over from the 24GB page: MoE saves compute and bandwidth, not weight memory. All 118B of Laguna’s parameters stay resident, because the router can call any expert on any token. The saving on this page is a different one, and it comes from the layer count, not the expert count.

The KV Cache Math

The formula, from how much VRAM for 128K context:

bytes/token = 2 (K and V) x layers_that_grow x kv_heads x head_dim x bytes_per_value

The phrase layers that grow is the whole page. A sliding-window layer holds only its window, 512 tokens for Laguna and 128 for gpt-oss, no matter how long the context is. Only full-attention layers grow with the window.

The dense 70B needs 6.7x the cache of Laguna and 8.9x the cache of gpt-oss for the same window. That ratio does not move with quantization, hardware or runtime. It is baked into the architecture.

What That Means on a 48GB Card

Take ~46GB usable after driver overhead. Check nvidia-smi; it varies.

This is the number the 48GB tier page raised and deliberately stopped at. The 48GB card was bought to run a 70B. It runs the 70B at a context length that most agent workloads exhaust in the first few tool calls. The MoE at the same footprint runs a full-length session on the same card, in the worst case with three times the window, in the best case with seven times.

The lever for the dense model is KV cache quantization. At q8_0 the 70B gets ~22–39K tokens. That is a usable session. It is still a fraction of what the MoE gets without touching a flag.

What That Means on a 96GB Card

At ~94GB usable the constraint lifts on both sides, and the comparison becomes a quality-versus-speed question rather than a fits-or-not question. The 96GB tier page has the full table. The relevant rows:

96GB is the first tier where a dense 70B runs the way its model card describes. It is also the tier where a 120B MoE has so much headroom that you can run a second model beside it.

Speed: A Ceiling You Can Derive

Token generation is bound by memory bandwidth. Every token requires reading the active weights once. So the ceiling is bandwidth divided by bytes read per token, before any overhead.

These are ceilings, not measurements. Real throughput is well below both, and the MoE’s gap to its ceiling is larger because expert routing, the shared expert and attention overhead do not scale down with active parameters. But the direction is not in doubt. The dense 70B on a single 3090 is a 10–20 tok/s experience. The MoE at the same footprint is a small-model experience. On a dual-3090 build the dense model also pays a PCIe hop between halves; the MoE at Q2 fits on the pair with room.

Quality: The Honest Version

Llama 3.3 70B is a December 2024 release. It remains the reference dense 70B in the open-weight world because nothing dense at that size has clearly replaced it. Laguna S 2.1 shipped in July 2026 and was built for agentic coding; its published Terminal-Bench figure is on our Laguna S 2.1 setup page. gpt-oss 120B is the model with the cleanest tool-call output we have found for agent loops.

So the comparison is not a fair fight on the leaderboard, and we will not pretend the 70B wins it. What the dense 70B retains:

• Predictable quality per token. Every token gets the full network. No routing variance.
• A quantization that goes lower gracefully. A dense 70B at Q4_K_M is a known quantity. A MoE at Q2 is a real quality drop; the Laguna page says so.
• Short-context tasks where 19K tokens is plenty. Classification, extraction, single-turn generation.

If your workload is the third one, the 70B at 48GB is fine and the whole context argument does not apply to you.

The Thing Sliding-Window Attention Takes Away

Here is the caveat no comparison prints.

When a sliding-window layer discards key-value entries outside its window, they are gone. They cannot be recovered without recomputing them. So on those layers there is no prefix cache, no context shift and no context reuse between requests.

For a chat, that is invisible. For an agent loop that re-sends a 6,000-token system prompt and a growing tool history on every step, it means the sliding layers re-process the prompt every turn instead of reusing it. llama.cpp’s --swa-full flag keeps the full cache on the sliding layers and restores reuse, at the memory cost in the table above: Laguna goes from 49,152 to 196,608 bytes per token, four times the cache.

So the choice at 48GB is sharper than “MoE gets more context”. It is: the MoE gets 130K tokens without prompt reuse, or 33K tokens with it. The dense 70B gets under 20K either way, but every one of them is reusable. Which of those three you want depends on whether your loop is long-context or repeated-prefix, and only you know that.

Hardware for the Tier Where the Choice Is Forced

Both models at Q4-class or Q2-class need 40GB or more of weight memory. That is the 48GB tier, and the two routes to it, priced as of August 2026:

EVGA GeForce RTX 3090 24GB — two of these, at $1,000–1,300 each used, are the cheapest 48GB. The MoE at Q2 fits the pair with room; the dense 70B fits with a PCIe hop between halves. Needs a 1200W supply and two spaced slots; see is NVLink worth it before you assume the bridge helps.

PNY NVIDIA RTX A6000 48GB — one card, one contiguous pool, 300W, $2,600–3,800 used. The single pool matters more for the dense 70B, which has no slack to split. Our default at this tier when the price lands low; details in best local LLM on an RTX A6000.

If you can reach 96GB, the choice stops being forced and both models run properly. That is a different page and a different price.

The Decision

See Also

• MoE vs Dense on a 24GB Card — the same argument one tier down
• Is 48GB of VRAM Enough in 2026? — where the 40 GiB number first appears
• Is 96GB of VRAM Enough in 2026? — where both models finally fit with their context
• Laguna S 2.1 Local Setup — quant sizes and commands for the MoE
• How Much VRAM for 128K Context? — the KV cache formula
• KV Cache Quantization: q8 vs q4 — the dense model’s only lever
• llama.cpp MoE Offload Flags Explained — running a MoE that does not quite fit
• Best Local LLM for 96GB VRAM — the gpt-oss 120B pick in detail

Sources

• Llama-3.3-70B-Instruct config.json: 80 hidden layers, 64 attention heads, 8 key-value heads, hidden size 8192, max_position_embeddings 131072
• Laguna-S-2.1 config.json: 48 hidden layers with layer_types of 12 full attention and 36 sliding attention, sliding_window 512, 48 attention heads, 8 key-value heads, head_dim 128, 256 experts, 10 experts per token, max_position_embeddings 1048576
• gpt-oss-120b config.json: 36 hidden layers alternating sliding and full attention, sliding_window 128, 64 attention heads, 8 key-value heads, head_dim 64, 128 experts, 4 per token, max_position_embeddings 131072
• Published GGUF sizes: Llama 3.3 70B Q4_K_M 42.52GB and Q8_0 74.98GB (bartowski); Laguna S 2.1 UD-Q2_K_XL 39.7GB and UD-Q4_K_M 73.1GB (unsloth); gpt-oss-120b MXFP4 63.4GB (ggml-org)
• llama.cpp sliding-window KV cache behaviour and the --swa-full flag: ggml-org/llama.cpp PR #13194 (SWA support) and discussion #24543 (clarification on --swa-full), which state that out-of-window entries are discarded by default and that keeping them is required for cache reuse
• KV cache and throughput figures are our arithmetic on the fields above. Bytes-read-per-token for the MoE is an estimate from the active-parameter fraction; it is labelled as an estimate and it is a ceiling, not a benchmark.
• Prices as of August 2026 from our hardware price reference: used RTX 3090 $1,000–1,300 (ResalePrices, gpudojo); used RTX A6000 $2,600–3,800 (gpudojo, used-A6000 guides)