Evals for coding agents: know if your agent's work is good
By Albert Nahas, 9 min read, updated
An eval for a coding agent is a repeatable check of whether its work is good, not just whether it runs. The useful setup has four parts: a set of real tasks with known good outcomes, a written rubric for what "good" means in your codebase, a grader that is separate from the agent doing the work, and enough repeated runs to tell a reliable agent from a lucky one. Passing tests is one line of the rubric, never the whole thing.
I care about this because I let agents merge their own pull requests on a product I built alone. That only works if I can say, in writing, what a mergeable change looks like. Writing that down turned out to be most of the job.
The gap between "tests pass" and "I'd merge this"
Tests check what someone thought to test. A reviewer checks everything else: does the change solve the actual problem, does it fit how the codebase does things, is it the smallest change that works, did it quietly delete a check or skip a case.
Laurie Voss made this concrete in his AI Engineer talk. On a Cognition benchmark he cited, the pass rate fell from 87% to 54% once the bar changed to "would you merge this". A third of the "passing" work was work a maintainer would send back. He also pointed to teams producing 741% more code while shipping only about 30% faster. More output didn't turn into proportionally more shipped value. My read is that review and rework absorb most of the difference.
That's the gap evals have to measure. If your eval only runs the test suite, you're measuring the 87%.
Write down what good means
This is the human part, and it's the same skill I describe in what a product engineer does: deciding what right looks like before anyone builds anything. An agent can write code all night. It can't decide your standards for you.
A rubric is a short list of checks a reviewer would apply, each one answerable from the diff and the task. A starting version looks like this:
# Merge rubric
1. Solves the task as stated. Nothing in the task is left undone.
2. Tests: existing tests pass, and new behavior has a test that would fail without the change.
3. Minimal diff. No refactors, renames or formatting changes the task didn't need.
4. Fits the codebase. Uses existing helpers and patterns instead of new ones.
5. No weakened checks. No deleted tests, skipped assertions, widened types or swallowed errors.
6. Verified. Lint, typecheck and build pass; UI changes were rendered and looked at.
7. Safe. No secrets, no new dependencies, nothing touching money, auth or user data
without a human.
Verdict: MERGE, FIX (list findings), or REJECT.
Every line targets a mistake agents make. Rule 5 exists because "make the tests pass" sometimes means making the test weaker. Rule 3 exists because agents love to tidy things nobody asked them to touch.
Keep the rubric in the repo, next to the charter or merge policy the agent already reads. The PR grading guide uses the same idea with A, B, C and X grades.
Use a separate grader
The agent that wrote the code is the worst judge of it. It has the same blind spots it had while writing, and if you ask it whether it's done, it has every reason in its own context to say yes.
So the grader is a separate run with fresh context. It sees three things: the task, the diff and the rubric. It never sees the author's plan or reasoning, so it judges the change instead of agreeing with the argument for it. That's the same rule I use for reviewer subagents.
LLM-as-judge works well enough for this when the rubric is specific. "Is this code good?" gets you vague praise. "Does the diff delete or weaken any existing test? Quote the line if so" gets you something you can check. Ask for evidence with every finding, and ask for a structured verdict you can count.
Two cautions. First, check the judge against yourself. Grade 20 diffs by hand, have the judge grade the same 20, and read every disagreement. If it agrees with you most of the time and the misses are borderline cases, it's usable. If it waves through things you'd reject, fix the rubric before trusting any numbers. Second, don't let the judge see the reference solution as "the answer". Different but correct is still correct.
Golden tasks
A golden task is a real task from your history with a known good outcome. The best source is your own merged PRs: take the issue or ticket as the prompt, check out the commit before the fix, and keep the merged diff as a reference.
Twenty to fifty is plenty to start. Pick a mix:
- Small fixes that should take one or two files.
- Features that cross a few layers.
- Traps: tasks where the obvious change is wrong, or where the right answer is "don't change the code", and tasks that touch something the agent must escalate instead of doing.
Freeze them. A golden set that changes every week can't tell you whether the agent got better or the tasks got easier. Add new ones when you find a new failure, but don't edit old ones.
pass^k: reliability, not luck
One run tells you almost nothing. Agents are not deterministic, and a task that passes once may fail the next three times.
Two numbers matter. pass@k is the chance that at least one of k attempts succeeds. That's the right measure when a human picks the best of several drafts. pass^k is the chance that all k attempts succeed, and it's the right measure for anything that runs unattended, because nobody is there to pick.
If an agent passes a task 80% of the time, its pass^5 is 0.8 multiplied by itself five times, about 33%. At 90% per run it's about 59%. A number that looks fine for one run looks very different when the agent runs every day. Run each golden task at least three to five times and report pass^k next to the plain pass rate. The gap between them is your flakiness.
A practical setup with headless Claude Code
You don't need a framework. A shell script, git worktrees and two claude -p calls per run cover it.
For each golden task and each attempt, create a clean worktree at the starting commit and run the agent with the same limits it has in production:
git worktree add "../eval/$TASK-$RUN" "$BASE_SHA"
cd "../eval/$TASK-$RUN"
claude -p "$(cat "tasks/$TASK.md")" \
--output-format json \
--max-turns 40 \
--max-budget-usd 3 \
--permission-mode dontAsk \
--allowedTools "Read,Edit,Write,Bash(npm test *),Bash(npm run lint)" \
> agent.json
git add -A && git diff --cached "$BASE_SHA" > change.diff
Then grade it in a separate run that gets only the task, the diff and the rubric, and can't edit anything:
cat change.diff | claude -p "You are reviewing a pull request.
Task: $(cat "tasks/$TASK.md")
Rubric: $(cat RUBRIC.md)
The diff is on stdin. Apply each rubric line, quote evidence for every finding,
and give a verdict." \
--output-format json \
--json-schema "$(cat grade.schema.json)" \
--permission-mode dontAsk \
--allowedTools "Read" \
> grade.json
With --json-schema, the verdict comes back in structured_output, so a script can count MERGE, FIX and REJECT across runs. The agent's JSON also has total_cost_usd, which belongs in the results table next to quality. An agent that gets 5% better at three times the cost may not be better for you.
Run the test suite yourself too, outside the agent, and record it as its own column. "Agent says tests pass" and "tests pass" are different facts.
The scheduling guide covers the flags in more detail, including --bare for runs that should behave the same on every machine.
Turn reviewer feedback into rules
The most valuable output of an eval isn't the score. It's the list of findings.
Every time the grader or a human sends a change back, ask whether the same mistake could happen again. If it could, it becomes a rule somewhere:
- In the rubric, if it's a judgment call the grader should check.
- In CLAUDE.md or a skill, if the agent just didn't know how the codebase does it.
- In a hook, if it must never happen. A PreToolUse hook that blocks edits to migration files beats a sentence asking nicely.
- In the golden set, as a new task that reproduces the mistake, so you'll know if it comes back.
This is how the rubric grows. It starts with five lines you thought of and ends up with fifteen lines your agents taught you.
Measure what ships
Offline evals tell you whether a change to your prompt, model or rules made things better before you roll it out. Production tells you whether it held up. Track a few numbers on the agent's real PRs:
| Metric | What it tells you |
|---|---|
| Share merged without human edits | The production version of "would you merge this" |
| Share sent back for fixes | How often the first attempt misses |
| Reverts within two weeks | Changes that passed review but were wrong |
| Cost per merged PR | Whether the quality is affordable |
Watch reverts most closely. A rising merge rate with rising reverts means the grader got softer, not the agent better.
What I'd do next
Write the rubric first, even before you have a grader. Then pull ten merged PRs into a golden set, run each one three times, and grade the results by hand. You'll learn more from that afternoon than from any benchmark. Once the judge agrees with you, wire it into your merge policy and put the rules it surfaces into the agent's charter.
Questions people ask
What is an eval for a coding agent?
A repeatable test of whether the agent's work is good. You run it on a fixed set of real tasks, grade each result against a written rubric with a grader separate from the agent, and repeat runs to measure reliability.
Why isn't passing tests enough?
Tests only check what someone thought to test. In a Cognition benchmark Laurie Voss cited, the pass rate fell from 87% to 54% when the bar became whether a maintainer would merge the change.
What is LLM-as-judge?
Using a language model to grade another model's output against a rubric. For coding agents, run the judge with fresh context, give it only the task, the diff and the rubric, ask for evidence per finding, and check its grades against your own before trusting it.
What is the difference between pass@k and pass^k?
pass@k is the chance that at least one of k attempts succeeds. pass^k is the chance that all k succeed. Use pass^k for agents that run unattended: an agent that passes 80% of the time has a pass^5 of about 33%.
How many golden tasks do I need?
Twenty to fifty real tasks from your own merged PRs is enough to start, with a mix of small fixes, features and trap tasks. Freeze them so scores stay comparable, and add a new task each time you find a new failure.
Can I run coding agent evals with Claude Code?
Yes. Run each task with claude -p in a clean git worktree with the same limits as production, then grade the diff in a separate claude -p run with --json-schema so the verdict comes back as structured output you can count.