Local LLM Coding Setup on Windows + NVIDIA: The Guide Mac Tutorials Skip (July 2026)

Bottom Line (July 2026)

• You do not need WSL or Docker. Ollama, LM Studio, and llama.cpp all ship native Windows CUDA builds. Community WSL+Docker setups exist for specialized builds; they are optional.
• Pick by how much control you want. Ollama is the fastest to a working model. LM Studio gives you a GUI to see what is happening. llama.cpp gives you the flags that make a small card punch above its weight.
• Verify the GPU is engaged before you tune anything. Watch Dedicated GPU memory in Task Manager. Single-digit tokens/sec means you are silently on CPU.
• 12–16GB cards are not shut out. MoE models like Qwen 3.6 35B-A3B run on them via expert offload — a 3060 12GB is community-reported at ~51–53 tok/s.
• Match your KV cache types. Community gotcha: --cache-type-k and --cache-type-v set to different types can silently stop using the GPU.
• Laptop 4090 is 16GB, not 24GB. Plan against the real number, and expect thermal drift on long runs.

Why Windows Users Keep Getting Lost

The complaint shows up constantly in local-LLM threads, and it is fair:

“bruh im on windows and idk what and how to setup it properly… with these tutorials im just lost”

“Please make a guide without vps/docker+wsl bs in windows”

Both are community-reported, and they describe a real gap. The people writing tutorials are largely on Apple Silicon, where unified memory changes the entire calculus. The people asking — the ones posting spec dumps hoping someone will tell them what to run — are overwhelmingly Windows gamers with an RTX 3060 12GB, a 4060 Ti or 5060 Ti 16GB, a 4090 laptop, or a 3090/4090 with 24GB.

Those are good machines for this. Nothing about the hardware is the problem. The problem is that the instructions were written for a different computer.

Step 1: Pick a Runtime

Three real options on Windows. All three are native — you install them like any other Windows app.

Start with Ollama or LM Studio. Get any model generating text on your GPU first. That proves your driver, your CUDA path, and your VRAM budget all work. Only then move to llama.cpp, and only if you need the offload flags.

Ollama on Windows:

winget install Ollama.Ollama
ollama run qwen3.6:8b

llama.cpp on Windows: download the prebuilt CUDA release from the project’s GitHub releases page, unzip it somewhere without spaces in the path, and run llama-server.exe from a terminal in that folder. No build toolchain needed for the standard CUDA builds.

You will see community setups that run llama.cpp inside WSL with Docker — usually to reproduce a Linux-only build such as turboquant-style quantization pipelines. That is a legitimate reason to reach for WSL. It is not a requirement for running a model, and it is not where you should start.

Step 2: NVIDIA Specifics

Update the driver first. Not the one that shipped with the card. New model architectures and new CUDA builds routinely need a recent driver, and “it loads but crawls” is a common symptom of an old one.

Get a CUDA build, not a CPU build. llama.cpp publishes several Windows binaries per release. The CPU-only one will run your model perfectly well and use none of your GPU. This catches people constantly.

Then verify the GPU is actually engaged. This is the step that separates a working setup from three days of confusion. Open Task Manager → Performance → your GPU, and watch Dedicated GPU memory while the model loads. It should climb by roughly the size of the model file.

nvidia-smi --query-gpu=memory.used,memory.total,utilization.gpu --format=csv -l 1

The fastest tell is your tokens-per-second number. Single-digit tok/s on a modern NVIDIA card means you are on CPU, whatever the logs imply. A card in this class running a sane quant should be in the tens.

The KV cache gotcha. Community-reported and genuinely nasty: set --cache-type-k and --cache-type-v to the same type. Mixing them — a quantized K cache against an f16 V cache — can silently drop the GPU path and leave you CPU-bound with no error printed. If your speed collapsed immediately after you touched cache flags, that is the first thing to undo.

--cache-type-k q8_0 --cache-type-v q8_0

Also worth knowing before you reach for those flags at all: f16 KV cache is faster than q8_0 whenever it fits. Quantized cache is a memory feature, not a speed feature.

Step 3: Fit the Model to a Gaming Card

A 12GB or 16GB card cannot hold a 35B dense model. It can run a 35B Mixture-of-Experts model, because only a few billion parameters are active per token — the expert weights can sit in system RAM while attention stays on the GPU.

That is what makes Qwen 3.6 35B-A3B viable on a 3060 12GB. Community-reported config and result: 32GB of DDR5, IQ4_NL quant, 64K context, roughly 51–53 tokens/sec.

llama-server.exe ^
  -m .\qwen3.6-35b-a3b-IQ4_NL.gguf ^
  -ngl all ^
  --n-cpu-moe 25 ^
  --flash-attn on ^
  --no-mmap ^
  -c 65536 ^
  --host 127.0.0.1 --port 8080

On a 16GB card (4060 Ti 16GB, 5060 Ti 16GB, 4080, laptop 4090) start around --n-cpu-moe 20 and sweep downward. The rule is: lower the number until speed collapses, then step back up one. That collapse is a VRAM spill into system memory, and on Windows it usually arrives without an error message — just a sudden 2-3x slowdown.

Do not copy anyone’s number, including these. Your quant, RAM speed, and context length are not theirs. The full explanation of each flag and the sweep method is in llama.cpp MoE offload flags explained.

One more thing that bites Windows users specifically: set your context size to what you actually need before you tune. The KV cache lives in VRAM, so a context window you never use is expert layers you could have had on the card.

Step 4: Wire a Coding Agent to It

Both Ollama and llama.cpp’s llama-server expose an OpenAI-compatible endpoint. That is the universal adapter — most coding harnesses will accept a base URL and a model name.

:: llama-server
http://127.0.0.1:8080/v1

:: Ollama
http://127.0.0.1:11434/v1

In practice you set two environment variables and point the tool at your box:

setx OPENAI_BASE_URL "http://127.0.0.1:8080/v1"
setx OPENAI_API_KEY "local"

Open a new terminal after setx — it does not affect the one you typed it in.

For Claude-Code-style tools that speak the Anthropic message format rather than the OpenAI one, llama.cpp has a native Anthropic-style /v1/messages endpoint (community-reported). That removes the need for a translation proxy in the middle, which used to be the standard workaround and was a common source of tool-calling breakage.

Expect tool calling to be the shakiest part of a local setup regardless of runtime — it is the capability that degrades first at small sizes and low quants. Local LLM tool calling reliability covers which failures are the model’s fault and which are the harness’s.

Laptop GPU Caveats

Laptop cards carry the same names as desktop cards and different hardware.

• A laptop RTX 4090 has 16GB, not 24GB. Plan your quant against 16.
• Power limits are real. The same chip at 80W and at 150W are different products. Plug in, and set the Windows power mode to Best performance before benchmarking.
• Thermals drift. Your first 30 seconds of generation is not your sustained speed. Run a long generation before you trust a number.
• Hybrid graphics. Confirm the discrete GPU is the one doing the work, not the integrated one. Task Manager lists both.

Windows Troubleshooting

Related Guides

• llama.cpp MoE offload flags explained — the flags and the sweep method behind the 12–16GB configs above
• Best local LLM for the RTX 3060 12GB and RTX 4060 Ti 16GB — model picks for the most common Windows cards
• Best local LLM for the RTX 3090 and RTX 4090 — the 24GB tier
• Ollama vs llama.cpp — when the extra control is worth leaving Ollama
• Why local LLMs are slow even when they fit — the silent spill failure mode
• OpenClaw Windows setup — wiring the agent side on Windows
• Local LLM tool calling reliability — why agent loops break on small local models