Claude Code Commands, Part 4: build-eval and hillclimb for Your Claude App

Final part. Parts 1 to 3 were about how you work with Claude Code; Part 4 is about the Claude-powered app you're building with it. New running example, also invented: a meme-caption vibe classifier. You paste a meme caption, the app calls Claude, and it returns one label plus a flag for when a human should take a look. Here's its whole output contract:

{
  "vibe": "wholesome" | "cursed" | "cringe" | "other",
  "needs_review": true | false
}

"It feels better since I changed the prompt" is not a metric. Two /claude-api workflows turn that feeling into a number: build-eval builds the eval set, and hillclimb improves the app against it. Both need Claude Code v2.1.259 or later. The facts here come from Anthropic's Skills docs, Lance Martin's claude.dev post on automating eval design and hillclimbing, and Anthropic's "Demystifying evals for AI agents".

TL;DR

• /claude-api build-eval interviews you, samples real inputs, proposes graders and validates results with confidence intervals.
• A trustworthy eval mirrors production, scores higher with stronger models, leaves headroom below 100% and has low variance.
• /claude-api hillclimb proposes one change per round, scores it on a train/test split, and reverts gains that only show up in training.
• Anthropic reports a customer-support eval going from 78.6% to 90.5% accuracy on held-out tickets at about a fifth of the cost per ticket.
• Training score up, held-out flat? That's overfitting, not progress. 🚩

build-eval: let Claude interview you

Run /claude-api build-eval in the app's repo. The Skills page sums it up as "Build an eval set for your Claude-powered app". Lance Martin's post fills in how: it interviews you about the app, samples inputs from wherever real ones live (production data, bug reports), proposes graders, and validates the results with confidence intervals. What you get back is a graded evaluation you can run again and again.

Feed it real material. Anthropic's evals guide suggests starting with 20 to 50 realistic tasks drawn from real failures. For the vibe classifier, that means captions users actually reported as mislabelled, not ones we dream up at 2 a.m. A few illustrative cases to show the shape:

{"caption": "my plant survived the week 🌱", "expected": {"vibe": "wholesome", "needs_review": false}}
{"caption": "POV: the group chat at 3am", "expected": {"vibe": "cursed", "needs_review": false}}
{"caption": "me explaining my startup to grandma", "expected": {"vibe": "cringe", "needs_review": false}}
{"caption": "[a screenshot of a news headline]", "expected": {"vibe": "other", "needs_review": true}}

Pick graders that fit the job

A grader is whatever decides pass or fail. "Demystifying evals for AI agents" describes three kinds, and a decent eval usually mixes them:

Is your eval lying to you?

An eval that always says yes is worse than having none, because it makes you confident. Lance Martin's post lists four properties of an eval you can trust:

The red flags, from the same two sources: ambiguous graders and tasks that always fail point to a miscalibrated eval, not a dumb model. Read the transcripts, not only the scores. And isolate trials with a clean environment for every run, because shared state causes correlated failures. In the vibe classifier, if one case's cached response leaks into the next, ten red rows might really be one bug.

hillclimb: one change per round

With a baseline score in hand, /claude-api hillclimb iteratively improves your app against the existing eval. It splits the data into train and test sets, proposes a single change per round, and reverts changes that raise training scores without a test gain. That last rule is the whole point: a win that only shows up on cases the optimizer has seen isn't a win.

On top of what the skill does, we add three house rules. They're our habits, not features of the command:

• One file it may edit. Say which surface is fair game (for us, src/lib/vibePrompt.ts), so a "win" can't come from quietly touching the grader or the cases.
• A decision log. Change, training score, held-out score, kept or reverted. Future you will want to know why round 3 got binned.
• Stop rules. Stop after a few rounds with no held-out gain, when you hit the token budget you set, or when the eval runs out of headroom. Then improve the eval instead of the app.

round  change                                    train  held-out  verdict
1      define each vibe in one plain sentence    +6     +4        keep
2      add rule: any emoji means wholesome       +5     +0        revert
3      set needs_review when two vibes compete   +2     +3        keep

Three routes through the whole series

Four parts, a lot of commands. Here's how they chain together for the three kinds of work you're most likely doing:

The one rule across all four parts

Every command in this series is a bigger hammer than the one before it, and none of them replaces knowing what "done" looks like. Reach for the lightest command that still hands you something you can verify, whether that's a passing test, a screenshot or a held-out score, and only scale up when that stops being enough. The question Thariq Shihipar and Sid Bidasaria ask about workflows works for everything here: does it really need more compute?

It's how we run this blog, too. Our pipeline is an eval loop in disguise: the research file is the ground truth, a separate reviewer agent is the grader, and nothing gets committed until the checks have actually run with their output in the transcript. Same idea, smaller stakes than a support bot. 🫡

The whole series

1. Part 1: Audit your prompts with /claude-api prompt-audit
2. Part 2: /effort, /simplify and /frontend-design
3. Part 3: ultracode, /batch, /compact and /rewind
4. Part 4: build-eval and hillclimb for your Claude app (this post)