Context Window Traps (July 2026): Why Your Local Agent Breaks After 10 Prompts

Bottom Line (July 2026)

• The default is the bug. Runtimes ship a conservative context length, not the model’s native window. A model advertised at 1M tokens will happily run in a 4K window and never tell you.
• Agents pay 15–20K tokens before you type. System prompt plus one JSON schema per tool. In a 4–8K window there is no room left for a conversation, and the truncation eats the tool definitions first.
• “The model went dumb” is almost always truncation. Looping, forgetting it has tools, claiming work it did not do, or the harness spawning a new session every few minutes.
• Context costs VRAM, linearly, on top of the weights. And the cost per token is architecture-specific — community testing repeatedly finds Gemma’s KV cache far heavier per token than Qwen’s at similar sizes.
• Every tok/sec number you have read is a shallow-context number. One 12GB owner reports throughput settling to about 30 t/s once past 36K tokens. Speed decays as the window fills.
• KV quantization saves memory, not time. Community testing reports f16 beating q8_0 whenever f16 fits.
• Sizing: chat 8–16K. Coding agent 64K minimum, 128K comfortable.

The Symptoms

These are real reports, and they all have the same cause:

“openclaw with local install breaks after ~10 prompts — at the moment there is no way to use local models”

“The models are limited to 4k context… Hermes keeps spawning a new session every 5 minutes”

“Hermes Agent has a minimum requirement of 64,000 tokens… lowering runtime context to 16k/32k still causes Hermes to refuse execution” — on a 16GB Mac

“Why would you only have 32k context set when the model is meant for 1M?! That’s like a Ferrari with a 32mph limit”

“MacBook m4 48GB, asked 2+2, took 20 seconds — what’s wrong?”

Nothing is wrong with the model in any of these. Four of the five are a context window set below what the harness needs, and the fifth is the prefill cost of the harness’s own prompt.

Trap 1: Your Runtime Is Not Using the Model’s Context Window

The number on the model card is what the architecture supports. The number in your runtime is what you actually got, and they are rarely the same.

Ollama picks a conservative default because it has to load on an 8GB laptop without failing. Older builds defaulted to 2K; recent builds default higher, but still well below native maximums on large-context models. It does not warn you. It does not error. It just quietly runs a Ferrari at 32mph, which is exactly what that community complaint is describing.

Check what you actually loaded:

ollama ps

The output includes the context length of the running model. If that number is smaller than what your agent needs, you have found your bug and can stop reading the rest of this section.

Set it globally on the server:

OLLAMA_CONTEXT_LENGTH=65536 ollama serve

This applies to every model the server loads. It is the setting most people actually want, and the one most people never set.

Set it per request:

curl http://localhost:11434/api/chat -d '{
  "model": "qwen3.5:32b",
  "messages": [{"role":"user","content":"hello"}],
  "options": { "num_ctx": 65536 }
}'

Note that num_ctx is inside options. Put it at the top level and it is ignored silently — a common and very frustrating mistake.

Bake it into the model so you cannot forget:

FROM qwen3.5:32b
PARAMETER num_ctx 65536

ollama create qwen-agent -f ./Modelfile

llama.cpp uses -c, and -c 0 means “use the model’s trained maximum,” which on a 1M-context model will try to allocate a KV cache you do not have:

llama-server -m model.gguf -c 65536 --flash-attn on

LM Studio exposes it as “Context Length” in the model load settings, and it must be set before the model loads. Changing it in a running chat does nothing until you eject and reload.

Trap 2: The Agent Spends Your Window Before You Do

This is the part that surprises people who have only used local models for chat.

When you send a chat message, the payload is your message plus a short system prompt. Maybe a few hundred tokens. An 8K window feels enormous.

When an agent harness sends a message, the payload is:

1. A long behavioral system prompt — how to plan, when to use tools, output formatting, safety rules.
2. A full JSON schema for every registered tool. File read, file write, shell, search, browser, MCP servers. Each one is a few hundred tokens of parameter definitions.
3. Any injected project context, memory files, or directory listings.
4. Then, finally, your message.

Community reports put steps 1 through 3 at 15–20K tokens before the first user turn. Some harnesses declare the requirement outright — the Hermes report above cites a 64,000-token minimum and refuses to execute below it, which is honest behavior compared to runtimes that just truncate.

Now put that in a 4K window. There is no room for the tool schemas at all. The runtime drops tokens from the front of the context to make room, and the front is exactly where the system prompt and tool definitions live. The model receives a conversation with no instructions and no tools, and behaves accordingly: it answers in prose instead of calling a tool, repeats itself, invents results, or the harness detects the broken state and starts a fresh session. Every five minutes, as reported.

The tenth prompt is not a magic number. It is roughly where system prompt plus tool schemas plus ten turns of file contents and tool output crosses a small window. Read three large files and it happens on turn four instead.

Trap 3: The KV Cache Is a VRAM Purchase

The context window is not free. Every token in it stores key and value tensors for every attention layer, and that KV cache lives in VRAM alongside the weights.

Two properties matter:

It scales linearly. Double the context, double the cache. There is no economy of scale, which is why “just set it to 1M” fails immediately.

The per-token cost is architecture-specific. It is driven by layer count, number of key/value heads, and head dimension — not by parameter count. Two models of the same size can differ by several times. Community testing consistently reports Gemma’s KV cache being far heavier per token than Qwen’s, which is why people swap a Qwen model for a similarly sized Gemma one at the same context setting and suddenly get out-of-memory errors or a spill into system RAM.

Rough budgeting, and these are approximate and model-dependent — measure your own:

Ranges are order-of-magnitude guidance at f16, not a spec. The reliable way to get your number is to load the same model at two context lengths and diff the reported VRAM.

The practical consequence: on a 24GB card running a 20GB model, you have about 4GB for cache, which is 8–16K of context. That is a chat window, not an agent window. The upgrade path for agentic work is headroom, not a bigger model. Our 24GB vs 32GB breakdown walks through exactly that trade on one model.

Trap 4: Every Speed Number You Have Seen Is a Shallow-Context Number

Benchmarks quote tokens per second at the start of a conversation, when the KV cache is nearly empty. Attention cost grows with the number of tokens already in the window, so throughput decays as the conversation fills up.

One 12GB owner reports throughput settling to about 30 t/s once past roughly 36K tokens on a model that benchmarks much higher on a fresh prompt. That is not a regression — it is what the same model does at real agent context depth.

Read every advertised t/s number as a best case at turn one. For agent work, the number that matters is the one at 30K+ tokens, and almost nobody publishes it.

This also explains the “2+2 took 20 seconds” report on a 48GB M4. That is prefill, not generation. Before the model produces its first token it has to process the entire prompt, and inside an agent that prompt is the full system prompt plus every tool schema. The model spent 20 seconds reading its own instructions and about 50 milliseconds answering. Prompt caching or a trimmed tool set fixes it; a faster model does not, because generation was never the slow part.

Trap 5: KV Quantization Is a Memory Lever, Not a Speed Lever

When people hit the VRAM wall they reach for quantized KV cache, usually expecting it to be faster as well as smaller. It is not.

Community testing reports f16 KV cache running faster than q8_0 whenever f16 fits, because the quantized cache adds conversion work on every single token. You are trading compute for memory.

# llama.cpp — only when f16 does not fit
llama-server -m model.gguf -c 131072 --flash-attn on \
  --cache-type-k q8_0 --cache-type-v q8_0

The rule: use f16 while it fits. Reach for q8_0 when the alternative is a shorter window or a spill into system RAM, which costs far more speed than the quantization does. Skip q4_0 for anything where accuracy matters.

Symptom to Cause

Sizing Guide

Chat: 8–16K. A conversation and a couple of pasted files. This is what the runtime defaults are tuned for and it works fine.

Coding agent: 64K minimum. System prompt and tool schemas take 15–20K, which leaves roughly 45K for actual work. That is a handful of source files, a diff, and the conversation. Below this, you spend your session watching the agent re-read files it already read.

Comfortable agent: 128K. The agent holds a real working set across a multi-step task without dropping what it learned in step two. On consumer hardware this usually means a smaller or lower-quant model, and that is the correct trade — a 14B model with 128K of context outperforms a 32B model with 8K on agentic work, because the 32B one cannot see its own tools.

Set the window first, then pick the largest model that still leaves room for it. Most people do this in the opposite order, which is how you end up with a very good model that breaks after ten prompts.

Related Guides

• Ollama crash course for local agents — the rest of the runtime settings that matter
• Why local LLMs are slow even when they fit — the spill-to-RAM failure mode in detail
• 128GB local LLM context window — what a big-memory machine actually buys you
• OpenClaw + Hermes hardware requirements, honestly — what these agents really need
• Local LLM tool calling reliability — when it is the model, not the window
• OpenClaw out of memory — when the KV cache is what broke the load