Hybrid Routing: Which Tasks to Send to a Local LLM and Which to the Frontier (July 2026)

Bottom Line (July 2026)

• Selective, not full, replacement. The framing that keeps surfacing in local-AI threads: “the best economic benefit of local AI is selective and not full replacement — find something you do repeatedly that is expensive to do in frontier models.”
• The 90/10 split. “Use them for 90% and that last 10% just rent cloud.” Local handles the volume; the frontier handles the ceiling.
• Route on four axes: repetition x token volume, required capability, privacy, and failure cost. Three of the four push local; capability is the one that pulls frontier.
• Wiring is trivial. Both llama.cpp’s llama-server and Ollama speak the OpenAI API shape, so switching a harness between local and cloud is a base URL and a model name.
• Escalate on evidence. Two failed local attempts on the same subtask, or a context requirement past your KV cache budget, means hand it to the cloud.
• The honest caveat: small models fail hard when over-scoped. “Build a Twitter clone” will fail horribly; “refactor this class” is in line with their capabilities.

The Rule Nobody Writes Down

Most local-LLM content answers “which model fits my GPU.” Almost none of it answers the question people actually hit on day three: given that I now have a local model, what do I actually send it?

The answer is not a model comparison. It is a routing policy. And the shape of that policy shows up independently across every local-AI community — people arrive at the same conclusion without coordinating. One version: “have your local model handle the majority of tasks, but anything with a complicated script it should create an agent using a cloud model.” Another: “a smart router that uses local hosted then cloudbursts to cloud providers when it needs capability.” A developer with 22 years in the field described rotating models per job — planning and architecture to a frontier model, wide-context bug hunts to a big-context model, local for quick edits.

That is the same policy three times. Here it is written out.

The Four Routing Signals

Score a task on four axes. Three of them push work local. One pulls it to the frontier.

The useful property of this scoring is that signal 1 and signal 2 usually point in opposite directions, and that is fine. High-volume work is rarely high-capability work. Summarizing a hundred transcripts is a lot of tokens and very little thinking. Designing a migration is very little text and a lot of thinking. The split is natural.

When signals conflict — a high-volume task that also touches production — failure cost wins. Run it on the frontier until you have measured the local model’s error rate on that exact job.

What Goes Local

These are the jobs where per-token cloud pricing hurts the most and where a mid-size local model is genuinely good enough.

• Summarization at volume. Meeting notes, transcripts, long email threads, RSS digests. High tokens in, few tokens out, quality ceiling is low.
• Transcript and text cleanup. Filler removal, speaker labeling, punctuation repair on ASR output. Mechanical work.
• Classification and tagging. Ticket triage, sentiment, routing labels, spam filtering. Constrain the output to a fixed label set and small models are reliable.
• Boilerplate generation. Test scaffolds, config files, type definitions, CRUD handlers, docstrings.
• RAG preprocessing. Chunking, metadata extraction, synthetic question generation, embedding pipelines. This is the single biggest token sink in most RAG systems and almost none of it needs frontier reasoning.
• Single-file edits. Rename, extract a function, add error handling, convert a loop. Scoped to one file, verifiable by eye.
• Anything sensitive. Client data, health records, internal financials, unreleased code, personal journals. Privacy alone is sufficient reason regardless of the other three signals.

What Stays on the Frontier

• Multi-file refactors. Changing an interface and chasing every call site requires holding the repo’s shape in working memory. This is where local models fall off a cliff.
• Architecture and planning. The step where you decide what to build. Cheap in tokens, expensive to get wrong, and the quality difference is largest here.
• Hard debugging. Especially the wide-context kind: a bug whose cause is three modules away from the symptom.
• Anything customer-facing that ships unreviewed. Failure cost dominates.
• Novel or unusual domains. If the task is far from what the local model saw in training, the gap widens.

A practical pattern: use the frontier model to write the plan, then hand each step of that plan to the local model to execute. The expensive model does the thinking; the free model does the typing. That inverts the usual cost curve, because planning is short and execution is long.

The Wiring

The reason hybrid routing is practical in July 2026 is that everything speaks the same API.

Start a local OpenAI-compatible endpoint. llama.cpp’s llama-server exposes one directly:

llama-server -m ./models/your-model-Q4_K_M.gguf \
  --host 127.0.0.1 --port 8080 \
  -c 32768 --n-gpu-layers 999

Ollama does the same on port 11434:

ollama serve
# OpenAI-compatible base URL: http://localhost:11434/v1

Point your harness at it. Because the shape matches, switching a tool between local and cloud is two environment variables:

# local
export OPENAI_BASE_URL="http://localhost:8080/v1"
export OPENAI_API_KEY="sk-local"   # ignored, but most clients require a value

# frontier
unset OPENAI_BASE_URL
export OPENAI_API_KEY="sk-your-real-key"

Most agent harnesses — OpenClaw included — let you set a base URL per profile, so you can keep a local profile and a cloud profile and switch per session rather than per install. See why OpenClaw defaults to Claude and not Ollama if your local config appears to be ignored.

Use a cheap aux model for the background jobs. This is the pattern that surfaced repeatedly in Hermes agent threads: agent harnesses do a lot of work that is not the main reasoning loop — vision/OCR passes, context compression and summarization, title generation, memory writes, tool-output condensation. Those calls are frequent, and on a frontier model they quietly dominate the bill. Point the aux/small-model slot at your local endpoint and leave the primary model on the frontier. It is the highest-ratio change available: near-zero quality impact, large cost reduction. The real token cost of running a Hermes agent has the breakdown of where those calls come from.

Escalation Triggers

A router is only useful if it knows when to give up locally. Concrete triggers, in rough order of how often they fire:

1. Two failed attempts on the same subtask. Not two failed runs of the whole job — two failures on one step. Retrying a third time with the same model almost never works; the model is out of depth, not unlucky.
2. Context requirement exceeds your KV cache budget. If the files needed to answer do not fit in your local window, you will get a confident wrong answer rather than an error. Route it out before it starts.
3. Tool calling breaks down. Malformed JSON arguments, invented tool names, or repeated calls to the same tool. Local tool-calling reliability varies a lot by model — see local LLM tool calling reliability.
4. The model claims success without evidence. A known local-model failure mode: reporting a file was edited when nothing changed. If your verification step fails, escalate rather than re-prompt. More on this in when a local model says it did it but didn’t.
5. The output is going to production. Failure cost overrides everything upstream.

Log every escalation with the task type. After a couple of weeks that log is your real routing table — better than any rule you write in advance, because it reflects your model on your work.

The Honest Caveat: Scope Is the Failure Mode

The most common reason people conclude “local models are useless” is that they tested them with a frontier-sized prompt.

The cleanest statement of this came from a commenter describing where the line sits: “Build a Twitter clone” will fail horribly, while “refactor this class” is in line with their capabilities. Same model, same day, opposite outcomes. The variable is scope, not quality.

So when a local run fails, check the prompt before you check the model. If the task requires the model to decide what to build, choose an architecture, and then write it across a dozen files, that is a frontier task no matter how good the local model’s benchmark scores look. Narrow it until the unit of work fits in one or two files with a clear success condition, and the same model often clears it on the first try.

This is also why the plan-on-frontier, execute-on-local pattern works so well. It is not a cost trick. It is scope control — the plan is what turns a frontier-sized task into a sequence of local-sized ones.

Start Here

If you are setting this up today, do it in this order rather than trying to build a router first:

1. Find one repeated expensive job. Look at your last month of API usage and find the task you ran the most. That is your first candidate, not whatever is most fun to move.
2. Move only that one job local. Measure the error rate against what the frontier model produced for the same inputs.
3. Point your harness’s aux/small-model slot at local. Cheapest win available.
4. Add the two-failure escalation rule. Manually at first — you re-run it in the cloud yourself.
5. Automate the router last, using the escalation log you have accumulated.

Most people do this backwards: build the router, then discover they had nothing worth routing. The routing policy is the product of measurement, not a prerequisite for it.

Related Guides

• Zero-dollar OpenClaw with local models — the fully-local end of the spectrum
• Cut OpenClaw API costs — cloud-side savings that stack with hybrid routing
• The real token cost of a Hermes agent — where background calls actually go
• Local LLM tool calling reliability — the most common escalation trigger
• Ollama vs llama.cpp — picking the local server
• What local LLM fits my machine — sizing the local half