Harness Tests Explained: Check Your CLAUDE.md and Skills

Bottom Line

• Instruction files do not help by default. An ETH Zurich study found LLM-written context files made agents slightly worse and about 20% more expensive.
• Test two things for every skill: does it trigger on the right prompts, and does the result improve when it does?
• claude plugin eval tests plugins with and without the plugin. It does not load your CLAUDE.md.
• Hooks are the easiest to test. Pipe saved JSON into the script and check the exit code. No model call needed.
• OpenClaw has readiness checks (openclaw skills check, openclaw doctor --lint), not behavior tests.

Why Instruction Files Need Tests

The paper “Evaluating AGENTS.md” (Gloaguen and others, arXiv, version 3 from September 2026) tested context files with Claude Code, Codex and Qwen Code:

The agents did follow the instructions. The instructions just did not help much, and they made the agent take more steps. Repository overviews did not help at all. The authors found context files useful for practices that are not standard, and say changes “should be rigorously evaluated.”

What this means for you: a rule you add can cost tokens on every task and help on none. You only know if you measure.

What to Test

Test 1: Skill Triggers

A skill only helps if the agent loads it. Claude decides from the skill’s description, and Claude Code cuts description plus when_to_use at 1,536 characters in the skill list. Put the main use case first.

Write two lists of prompts:

• Should trigger: real phrasing a user would type, not the skill’s own name.
• Should not trigger: close but wrong requests.

Anthropic’s skill-creator plugin uses 20 such queries, splits them 60/40 into train and test sets, runs each 3 times, and tries up to 5 description rewrites. It picks the final description by the test score, so it does not overfit to the prompts it was tuned on. OpenAI’s Codex guide uses the same idea: a CSV with id,should_trigger,prompt and 10 to 20 prompts per skill.

How much this matters: Scott Spence measured Claude Code skill activation on 22 prompts (February 2026). With no help, skills fired 50% to 55% of the time. With a hook that forced an evaluation step, they fired 100%.

Check one result by hand. A GitHub issue (#79998) reports that the skill-creator trigger eval can read near 0% whatever the description says, because test probes register as slash commands. It was closed as stale, not fixed.

Test 2: Plugins With claude plugin eval

Claude Code v2.1.269 added claude plugin eval. It shipped in the week of 7 to 11 September 2026.

claude plugin eval init     # proposes prompts that should and should not trigger, and graders
claude plugin eval .        # runs the suite from the plugin root

Each case is a folder:

evals/<case>/prompt.md          # the prompt, with options such as max_turns
evals/<case>/graders/*.md       # one file per grader

There are six grader types:

The key feature is the baseline. Each case runs 3 times with the plugin and 3 times without. The table shows WITH, W/OUT and Δ. The docs put it plainly: if a case scores 1.0 with and without the plugin, the plugin is not what made it pass.

The most common first result, per the docs, is a Δ near zero with the Skill grader failing. That means Claude does not choose your skill on natural phrasing. Fix the description and run it again.

For CI, the docs give this pattern:

claude plugin eval . --trust-plugin --json results.json --threshold 0.8 \
  --model claude-sonnet-5 --judge-model claude-haiku-4-5 --no-publish --max-cost-usd 20

Pin the model, so that a model update is not mistaken for a plugin regression. Exit code 0 means the suite passed, 1 means a case fell below the threshold, and 2 means the cost limit stopped the run.

Two limits:

• It costs money. Every run and every judge grader is a real model call.
• It loads only the plugin. The docs say your CLAUDE.md files, hooks, MCP servers and other skills are absent.

Test 3: CLAUDE.md and AGENTS.md

Because claude plugin eval leaves CLAUDE.md out, test it with golden tasks:

1. Pick 5 to 10 real tasks from your repo history, each with a check you can run, such as a test that must pass.
2. Run each task with and without the file. Use a fresh session each time, with claude -p for Claude Code or codex exec --json for Codex.
3. Compare success, steps and tokens. Keep a rule only if it moves one of them.

promptfoo can automate this. Its anthropic:claude-agent-sdk provider loads no CLAUDE.md by default. Set setting_sources to project to load it, and compare two configs. It also has a skill-used assertion. Turn caching off when you test config changes, or the cache can hide them.

Anthropic’s skills docs give the reason for fresh sessions: context left from writing the skill “will mask gaps in the written instructions.”

Test 4: Hooks

Hooks are plain scripts with a clear contract, so they are the cheapest part to test.

• Input: for command hooks, “input arrives on stdin” as JSON, with fields such as hook_event_name, tool_name and tool_input.
• Output: exit 0 to proceed, exit 2 to block with a reason on stderr. The docs: “If your hook is meant to enforce a policy, use exit 2.”

A fixture test:

# tests/hooks/rm-rf.json holds a saved PreToolUse input for "rm -rf /"
./.claude/hooks/block-dangerous.sh < tests/hooks/rm-rf.json
test $? -eq 2 && echo "PASS: blocked" || echo "FAIL: not blocked"

Save one input that the hook must block and one it must allow. Run them on every commit. They need no model and cost nothing.

Test 5: OpenClaw

OpenClaw’s commands check setup and readiness. None of them score behavior.

Use skills check first when a skill “does nothing”. It is often not visible to the agent at all. Then test behavior with golden tasks, as for CLAUDE.md.

When to Run Which Test

Honest Caveats

1. The ETH results are averages. A file with rules your repo really needs can still help a lot. Measure your own.
2. Small test sets are noisy. With 10 tasks, one task is 10 points. Run each case more than once.
3. We did not run these tools for this guide. Commands and numbers come from the docs and posts below.
4. Hook fixture testing is a community method. It follows from the documented stdin contract, but the docs do not describe it.

Sources

• Gloaguen et al.: Evaluating AGENTS.md (v3, September 2026)
• Claude Code: plugin evals and week 37 release notes
• Claude Code: skills and hooks
• Anthropic skill-creator and Agent Skills best practices
• GitHub issue #79998: skill-creator trigger eval reads near 0%
• OpenAI: Testing agent skills systematically with evals
• Scott Spence: measuring Claude Code skill activation (February 2026)
• promptfoo: Claude Agent SDK provider
• OpenClaw docs: skills CLI, plugins CLI, doctor

See Also

• Harness engineering explained: what the harness is and why it matters
• AGENTS.md vs CLAUDE.md: one source of truth
• Your agent is ignoring CLAUDE.md: where it reads instead
• Claude Code hooks vs skills: enforce or instruct