Your Local Model Says It Wrote the File — It Didn't: Fixing Tool Calling for Local LLMs (July 2026)

Bottom Line (July 2026)

• The model is not lying to you. It never got a tool result back, so it completed the text with the outcome it expected. That is what completion models do.
• Cause 1, most common: broken chat template. The GGUF’s built-in template emits tool calls in a shape your harness cannot parse. For Qwen, load froggeric’s fixed chat templates. This is the fix the community converged on.
• Cause 2: context too small. Ollama’s default window silently truncates the system prompt that carries your tool schemas. Agent work needs 64K+.
• Cause 3: wrong sampler params. Default temperature causes thinking loops and malformed calls. Use the model author’s recommended values — Unsloth publishes them per model, and coding work generally wants temperature around 0.6.
• Cause 4: model or quant too small for the task. Structured output degrades before prose does. A step up in quant, or a smaller task, fixes more than a model swap.
• Cause 5: the harness itself. Same model, same backend, different frontend, different result. If one client loops forever, try another before blaming the weights.

What You Are Actually Seeing

Three symptoms, one underlying situation.

The literal tool call. The model’s reply ends with something like ...End of file. </tool_call: write_file>. The call text is sitting in the assistant message as prose. Nothing ran.

The confident narration. “I’ve created server.py with the routes you asked for.” No file exists. As one person put it after a week of this: “he says I did it, but he actually didn’t.”

The silent no-op. The agent runs its loop, produces a summary, and the working directory is untouched. “It never executes any command, it just prints them to me.”

All three are the same failure. The harness expects the model to emit a tool call in a specific structured format. It didn’t. The harness saw an ordinary text reply, so it did nothing and passed the text back into the conversation as if the turn were complete. The model, seeing no error and no tool result, continues the story.

This is worth internalizing because the instinct is to blame the model’s intelligence — and that sends you shopping for a bigger model when the actual problem is a template file. We have watched people conclude that a perfectly capable 30B model is “too dumb for agents” when it was a parsing mismatch the whole time.

Diagnostic Flow

Work top to bottom. The order is deliberate — the cheap checks catch most cases.

Cause 1 — The Chat Template

This is the one to check first and the one people check last.

A GGUF carries a Jinja chat template that decides how messages, tool schemas, and tool calls get rendered into the token stream. If that template’s tool-call syntax does not match what your inference server parses out, the call round-trips as plain text. Nothing errors. You just get prose where a function call should be.

Templates shipped in community quants have been wrong often enough that the fix became a shared resource. For Qwen models, the community converged on froggeric’s Qwen fixed chat templates on Hugging Face. One report we saw, on Qwen 3.6 35B A3B at Q4_K_M: “since that I have had almost no issues with tool calling.”

With llama.cpp, point at the fixed template file directly:

llama-server \
  -m models/qwen/Qwen-3.6-35B-A3B-Q4_K_M.gguf \
  --chat-template-file templates/qwen-fixed.jinja \
  --jinja \
  --host 127.0.0.1 --port 8080 \
  --ctx-size 65536 \
  --n-gpu-layers 999

--jinja matters. Without it llama.cpp falls back to a built-in template and your file is ignored.

With Ollama, the template lives in the Modelfile, so rebuild the model with the corrected one:

ollama show qwen3.6:35b --modelfile > Modelfile.qwen

# edit Modelfile.qwen: replace the TEMPLATE block
# with the contents of the fixed template

ollama create qwen3.6-fixed -f Modelfile.qwen
ollama run qwen3.6-fixed

Then point OpenClaw at the rebuilt tag rather than the original.

How to confirm this was your problem: run one trivial tool task (“list the files in this directory”). If the model now emits a real call instead of describing one, you are done — do not keep changing other settings.

Cause 2 — Context Too Small

Ollama’s default context window is small, and the failure mode is silent.

An agent harness front-loads a large system prompt: instructions, plus the full JSON schema for every tool it exposes. That block alone can be thousands of tokens before your first message. If the window cannot hold it, the tail gets truncated — and the tail is where the tool definitions usually sit. The model then genuinely has no tools. It answers in prose because prose is all it was given.

This looks identical to a template problem from the outside, which is why people chase it for hours.

OLLAMA_CONTEXT_LENGTH=65536 ollama serve

To make it stick across restarts on macOS:

launchctl setenv OLLAMA_CONTEXT_LENGTH 65536
# then restart the Ollama app

On llama.cpp it is --ctx-size 65536.

64K is the floor we would use for agent work, not a target. Whether you can afford it depends on what the weights left behind — see the context-window math. If raising the window makes generation crawl, your KV cache spilled to system RAM; that is a different problem, covered in why local LLMs are slow even when they fit.

Cause 3 — Sampler Parameters

Default sampler settings in inference servers are tuned for chat, not for structured output. Too much randomness and the model wanders off the exact token sequence a tool call requires, or gets stuck restating its reasoning without ever committing to an action.

Unsloth publishes recommended sampler parameters alongside their GGUF releases, per model. Use those rather than guessing. For coding and agent work the recommended temperature generally lands around 0.6 — low enough to hold format, high enough to avoid degenerate repetition.

llama-server \
  -m models/your-model.gguf \
  --temp 0.6 \
  --ctx-size 65536 \
  --host 127.0.0.1 --port 8080

In Ollama, set it on the model rather than per-request so the harness cannot override it with its own default:

ollama show your-model --modelfile > Modelfile.tuned

# add to Modelfile.tuned:
# PARAMETER temperature 0.6

ollama create your-model-tuned -f Modelfile.tuned

Check the model card before copying these numbers. Reasoning models and instruct models often want different values, and the author’s page is the source of truth — treat the 0.6 figure as a starting point for coding work, not a universal setting.

Cause 4 — Model or Quant Too Small for the Scope

Once template, context, and sampler are right, what remains is genuine capability. Two levers here, and neither is “buy a bigger model.”

Scope the task down. A model that reliably handles “read this file and tell me what the function does” may fall apart on “refactor this module across six files and update the tests.” Multi-step agent loops compound: every step is another chance to emit a malformed call, and one bad call derails everything after it. Give it one file and one operation at a time. This costs you turns and buys you reliability, and it is the single highest-leverage change most people can make without touching hardware.

Raise the quant. Heavy quantization degrades structured output before it degrades conversational fluency, because exact token sequences are more fragile than plausible-sounding prose. A model can chat well at IQ4_XS and still mangle JSON. One user’s fix for exactly this was moving from IQ4_XS to Q5 — plus adding a plain hint to the prompt telling the model to look up the correct tool-call format rather than assume it. That second half is worth stealing: an explicit format reminder in the system prompt costs almost nothing.

If you are choosing weights for agent work specifically, our best local models for OpenClaw guide ranks by tool-calling behavior rather than benchmark scores.

Cause 5 — The Harness

Same weights, same backend, different frontend, different outcome. This surprises people, but the harness owns prompt construction, tool-schema formatting, and tool-call parsing — three of the five things on this page.

We have seen reports of one client looping indefinitely on a task while open-webui, pointed at the identical backend, completed it. Others found Pi more reliable than what they started with. Nobody changed a model file in either case.

So before you conclude a model cannot do agent work, run the same task through a second harness. If it works there, the model is fine and you have narrowed the problem to prompt construction or parsing in the first client.

What Not To Conclude

Do not conclude the model is dishonest. “Lying about doing things while doing nothing” is a fair description of the experience and a bad description of the mechanism. There is no deception, only a missing tool result and a model completing text. The distinction matters because it points you at plumbing instead of at a model swap.

Do not conclude local models cannot do agentic work. Some genuinely handle it poorly. But the failures we see reported most often are configuration failures wearing a capability costume, and the fix is a template file and a context flag.

Do not change five things at once. Change one, run one trivial tool task, and see. Otherwise you will fix it and never know which change did it — and you will have to rediscover the whole thing on your next model.

Related Guides

• OpenClaw tool calling not working — the OpenClaw-specific version of this checklist
• Local LLM tool calling reliability — which models hold up in agent loops
• OpenClaw + Qwen 3.5 Ollama setup — a working config end to end
• OpenClaw defaults to Claude, not Ollama — check this before debugging anything else
• Ollama vs llama.cpp — template and sampler control differ sharply between them
• Best local models for OpenClaw — model-first picks for agent work