Pass Rate vs Completion Rate Explained: Agent Benchmarks
In September 2026, Google's Android Bench 2.0 said the best coding agent passed only about 28% of its long tasks. The same models score about 91% on the original tasks. The agents did not get worse. The tasks got much bigger, and pass rate counts a 95%-finished job as zero. This guide explains pass rate, completion rate and the other numbers on agent leaderboards, and how to check any benchmark claim before you pick a model.
Bottom Line
- Pass rate is all or nothing. A run counts only if every check is perfect.
- Completion rate gives partial credit. It measures how much of the real job was done.
- On long tasks the two numbers differ a lot. The top Android Bench 2.0 models pass 22% to 40% of long tasks, but complete more than 80% of the work.
- The number also depends on attempts, harness and who ran it. A different harness alone moved SWE-bench Verified scores by up to 15 points.
- For your agent, ask the question that fits the job. Do you need it done perfectly, done mostly, or done the same way every time?
The Android Bench 2.0 Story
Google published Android Bench 2.0 on 17 September 2026. It added long-horizon tasks that “take an engineer multiple days or even a week to complete”:
- upgrading dependencies,
- adding new features,
- building apps from scratch,
- converting a cross-platform app to Android.
One example task touches more than 125 files and 8,000 lines of code.
At launch, Google wrote that the best pass rate on these tasks was about 28%, against about 91% on the original tasks. The same models did both. On the leaderboard, Kimi K3 scores 90.2% on the original set and passes 12.0% of the long tasks.
The leaderboard of 26 October 2026 (30 long tasks):
| Model | Harness | Pass rate | Completion rate |
|---|---|---|---|
| Claude Sonnet 5.5 | Claude Code | 40.0% | 80.9% |
| Claude Opus 5.5 | Claude Code | 37.3% | 85.6% |
| GPT 6.1 Sol | Codex | 29.3% | 82.8% |
| GPT 6 Astra | Codex | 28.0% | 82.2% |
| Claude Fable 5.1 | Claude Code | 22.7% | 82.4% |
| Gemini 4 Argon | Antigravity SDK | 21.3% | 81.8% |
| Kimi K3 | Kimi Code | 12.0% | 72.8% |
Look at the top two rows. Sonnet 5.5 is first on pass rate. Opus 5.5 is first on completion rate. The two numbers rank the models differently.
How Completion Rate Is Scored
Google’s methodology page gives the formula: completion rate = base score x penalties.
The base score is a weighted sum of four parts. The task author sets the weights for each task.
| Part | What it checks |
|---|---|
| Functional | Do the feature tests pass? |
| Regression | Does the old behavior still work? |
| Requirements | Did it follow the instructions, library versions and architecture rules? |
| Visual | Does the layout match, with correct accessibility structure? |
The penalties multiply the score:
| Problem | Multiplier |
|---|---|
| The build fails | 0x |
| The agent cheats | 0x |
| A port reuses the old Flutter, Dart or JS files | 0x |
Legacy APIs, such as findViewById in Compose | 0.5x |
Pass rate is the share of runs that score a perfect 1.0. Each task runs 5 times, and the results are averaged.
Google’s best example of the gap: an upgrade of Bitwarden from Navigation 2 to 3, across 233 files, scored 0.409 completion and a 0.0% pass rate. Google’s own words: binary scoring rates such a run as 0%, “obscuring the model’s architectural capabilities.”
Two more findings from the launch post:
- No model reached a 100% pass rate on app ports. The best reached about 80% completion.
- Models did better at writing new code than at refactoring existing code.
Which Number Should You Care About?
| Your job | Number to read | Why |
|---|---|---|
| The agent works alone and must finish, such as a nightly migration | Pass rate | Partly done is broken |
| The agent drafts and you finish | Completion rate | 85% done saves most of your time |
| The agent runs many times on similar tasks | pass^k (all attempts succeed) | You need it to be reliable |
| You can retry and keep the best result | pass@k (any attempt succeeds) | One success is enough |
pass@1, pass@k and pass^k
These three names appear on most leaderboards:
- pass@1: one attempt per task. The share that succeed.
- pass@k: k attempts per task. The task counts if any attempt succeeds. This number goes up with k.
- pass^k: k attempts per task. The task counts only if all attempts succeed. This number goes down with k.
Real numbers show the size of the effect:
| Source | One attempt | More attempts |
|---|---|---|
| Codex paper (2021), pass@k | 28.8% | 70.2% with 100 attempts |
| tau-bench (2024), pass^k, gpt-4o | 61.2% | Under 25% when all 8 must succeed |
Anthropic’s evals guide gives the rule: use pass@k when one success matters, and pass^k when consistency is essential. It also gives the math: an agent that succeeds 75% of the time succeeds 3 times in a row only about 42% of the time (0.75 x 0.75 x 0.75).
Android Bench averages 5 runs. That is an average pass@1, not pass^5. A model with a 40% pass rate will not pass the same 40% of tasks every time.
The Harness Changes the Score
A benchmark tests a model inside a harness: the loop, tools and prompts around it. Change the harness and the score moves.
- Epoch AI (December 2025): on SWE-bench Verified, changing only the scaffold moved scores by up to 11 points for GPT-5 and up to 15 points for Kimi K2 Thinking. Epoch says the scaffold choice “has the single biggest impact on the overall performance.”
- An ICML 2026 workshop paper: on 50 Terminal-Bench Pro tasks, the harness changed pass rate by 0 to 8 points, but changed tokens per solved task by up to 40x.
Android Bench lists the harness on every row, and that is good practice. Our harness engineering page covers why this happens.
When a Benchmark Stops Being Useful
SWE-bench Verified is the main example. It has 500 human-checked GitHub issues.
- It is saturated. Vals.ai measured Claude Opus 5 at 97.0% on 1 September 2026, and stopped running it.
- Passing is not the same as mergeable. METR found that roughly half of the fixes that pass its tests would not be merged by the project maintainers.
- It may be in the training data. The Decoder reported that OpenAI stopped reporting it in February 2026, after models reproduced some original fixes from memory.
Harder benchmarks replace it. On Scale’s SWE-bench Pro, which uses copyleft and private code to limit training leaks, the top public score is 61.5%. On the new Terminal-Bench-Science, the top agent scored 30.0% on 70 tasks.
METR’s time horizon measures something else: the task length, in human time, that a model completes half the time. In January 2026, Claude Opus 4.5 led at 320 minutes, and the horizon doubled about every 89 days since 2024. METR notes that a 50% horizon does not mean the model completes 50% of tasks of that length.
A Checklist for Any Benchmark Claim
- Who ran it? The model maker, the benchmark team, or an independent lab? Epoch found that scores for the same model differ between API providers.
- Which harness? It can move the score by 15 points.
- How many attempts, and which metric? pass@1, pass@k or pass^k. Averaged over how many runs?
- Were all tasks run? Epoch notes that OpenAI ran 477 of 500 SWE-bench Verified tasks for o3 and o4-mini.
- Were turns or cost capped? Scale marks runs capped at 50 turns separately from 250-turn runs.
- Is the benchmark saturated or in the training data? Scores near 100% tell you little.
- Does “pass” mean usable? Check whether the tests can catch bad but passing code.
- Pass rate or completion rate? Pick the one that fits your job.
- What does a run cost? A model that passes 5 points more at 4x the cost may be the wrong pick for a nightly agent.
What This Means for Picking an OpenClaw Model
- Treat leaderboards as a short list, not an answer. Test the top 2 or 3 models on 5 of your own tasks.
- Run each task more than once. Note whether the agent succeeds every time, not only once.
- Measure in your harness. A score from Claude Code or Codex does not carry over exactly to OpenClaw.
- For local models, expect lower long-task pass rates and plan for a human to finish the job.
Our best local models for OpenClaw page ranks local models by tool calling.
Honest Caveats
- Leaderboards change quickly. The 28% launch figure was 40% six weeks later. Check the current board.
- Android Bench has 30 long tasks. One task is about 3.3 points of pass rate.
- We did not run these benchmarks. All numbers come from the sources below.
- Some OpenAI pages could not be read. Facts about OpenAI’s SWE-bench Verified decision come from news coverage, and we say so.
Sources
- Google: Android Bench 2.0 (17 September 2026), methodology and leaderboard
- Chen et al.: Evaluating Large Language Models Trained on Code (2021): pass@k
- Yao et al.: tau-bench (2024): pass^k
- Anthropic: Demystifying evals for AI agents (January 2026)
- Epoch AI: Why benchmarking is hard (December 2025)
- Vats and Golev, ICML 2026 workshop
- Vals.ai: SWE-bench Verified and METR: SWE-bench passing PRs that would not be merged (March 2026)
- The Decoder: OpenAI wants to retire SWE-bench Verified
- Scale AI: SWE-bench Pro public leaderboard
- Terminal-Bench 2.0 paper and Terminal-Bench-Science 0.1
- METR: Time Horizon 1.1 (January 2026)
See Also
- Harness engineering explained: why the same model scores differently
- Agent loops explained: verify before you call a task done
- Best local models for OpenClaw: local picks for tool calling
- Which coding agent should you use: a workflow decision