# Claude Code Commands, Part 4: build-eval and hillclimb for Your Claude App > Part 4 of our Claude Code commands series: build an eval with /claude-api build-eval, check it's trustworthy, then hillclimb on held-out cases. Source: https://bytevancer.com/blog/claude-code-commands-part-4-build-eval-hillclimb · Published 2026-10-03 · ByteVancer · AI Engineering **"It feels better" isn't a metric. Part 4: let /claude-api build-eval build the eval, check whether it's lying, then hillclimb one change at a time on held-out cases.** > **Part 4 of 4: Claude Code commands, explained** — [Part 1: Audit your prompts with /claude-api prompt-audit](https://bytevancer.com/blog/claude-code-commands-part-1-prompt-audit) · [Part 2: /effort, /simplify and /frontend-design](https://bytevancer.com/blog/claude-code-commands-part-2-effort-simplify-frontend-design) · [Part 3: ultracode, /batch, /compact and /rewind](https://bytevancer.com/blog/claude-code-commands-part-3-ultracode-batch-compact-rewind) · **Part 4: build-eval and hillclimb for your Claude app (you're here)** Final part. Parts 1 to 3 were about how you work with Claude Code; Part 4 is about the Claude-powered app you're building with it. New running example, also invented: a meme-caption vibe classifier. You paste a meme caption, the app calls Claude, and it returns one label plus a flag for when a human should take a look. Here's its whole output contract: *Our illustrative output contract, not from any Anthropic example.* ```text { "vibe": "wholesome" | "cursed" | "cringe" | "other", "needs_review": true | false } ``` "It feels better since I changed the prompt" is not a metric. Two /claude-api workflows turn that feeling into a number: build-eval builds the eval set, and hillclimb improves the app against it. Both need Claude Code v2.1.259 or later. The facts here come from Anthropic's Skills docs, Lance Martin's claude.dev post on automating eval design and hillclimbing, and Anthropic's "Demystifying evals for AI agents". ## TL;DR - /claude-api build-eval interviews you, samples real inputs, proposes graders and validates results with confidence intervals. - A trustworthy eval mirrors production, scores higher with stronger models, leaves headroom below 100% and has low variance. - /claude-api hillclimb proposes one change per round, scores it on a train/test split, and reverts gains that only show up in training. - Anthropic reports a customer-support eval going from 78.6% to 90.5% accuracy on held-out tickets at about a fifth of the cost per ticket. - Training score up, held-out flat? That's overfitting, not progress. 🚩 ## build-eval: let Claude interview you Run `/claude-api build-eval` in the app's repo. The Skills page sums it up as "Build an eval set for your Claude-powered app". Lance Martin's post fills in how: it interviews you about the app, samples inputs from wherever real ones live (production data, bug reports), proposes graders, and validates the results with confidence intervals. What you get back is a graded evaluation you can run again and again. Feed it real material. Anthropic's evals guide suggests starting with 20 to 50 realistic tasks drawn from real failures. For the vibe classifier, that means captions users actually reported as mislabelled, not ones we dream up at 2 a.m. A few illustrative cases to show the shape: *Illustrative cases written for this post (one JSON object per line). Your real ones should come from your users.* ```json {"caption": "my plant survived the week 🌱", "expected": {"vibe": "wholesome", "needs_review": false}} {"caption": "POV: the group chat at 3am", "expected": {"vibe": "cursed", "needs_review": false}} {"caption": "me explaining my startup to grandma", "expected": {"vibe": "cringe", "needs_review": false}} {"caption": "[a screenshot of a news headline]", "expected": {"vibe": "other", "needs_review": true}} ``` ## Pick graders that fit the job A grader is whatever decides pass or fail. "Demystifying evals for AI agents" describes three kinds, and a decent eval usually mixes them: **Grader types from Anthropic's "Demystifying evals for AI agents"; the last column is our example.** - **Code-based** — Good at: Fast and objective; Watch out for: Brittle; In the vibe classifier: Valid JSON? One of the four labels? Label matches expected? - **Model-based** — Good at: Flexible; Watch out for: Non-deterministic; needs calibration; In the vibe classifier: Was needs_review justified on a borderline caption? - **Human** — Good at: The gold standard; Watch out for: Slow and expensive; In the vibe classifier: Spot-check the cases where the other graders disagree ## Is your eval lying to you? An eval that always says yes is worse than having none, because it makes you confident. Lance Martin's post lists four properties of an eval you can trust: **Four signs an eval deserves your trust** - **Real tasks** Mirrors production — The cases look like what users actually send. - **Scales up** Rewards capability — Scores rise with stronger models or more effort. - **< 100%** Leaves headroom — Even frontier models don't max it out. - **Low variance** Stays stable — Repeat runs land close to each other. *The four properties from Lance Martin, claude.dev (Sep 28, 2026); the short labels are ours.* The red flags, from the same two sources: ambiguous graders and tasks that always fail point to a miscalibrated eval, not a dumb model. Read the transcripts, not only the scores. And isolate trials with a clean environment for every run, because shared state causes correlated failures. In the vibe classifier, if one case's cached response leaks into the next, ten red rows might really be one bug. ## hillclimb: one change per round With a baseline score in hand, `/claude-api hillclimb` iteratively improves your app against the existing eval. It splits the data into train and test sets, proposes a single change per round, and reverts changes that raise training scores without a test gain. That last rule is the whole point: a win that only shows up on cases the optimizer has seen isn't a win. **One hillclimb round, plus our logging step** Split → Propose → Score → Keep or revert → Log it 1. **Split** — Cases go into a training set and a held-out test set. 2. **Propose** — One change to the app per round. 3. **Score** — Run it on training and on held-out. 4. **Keep or revert** — Held-out gain: keep. Training-only gain: revert. 5. **Log it** — Our addition: one line per round, so you have receipts. On top of what the skill does, we add three house rules. They're our habits, not features of the command: - **One file it may edit. **Say which surface is fair game (for us, src/lib/vibePrompt.ts), so a "win" can't come from quietly touching the grader or the cases. - **A decision log. **Change, training score, held-out score, kept or reverted. Future you will want to know why round 3 got binned. - **Stop rules. **Stop after a few rounds with no held-out gain, when you hit the token budget you set, or when the eval runs out of headroom. Then improve the eval instead of the app. *Illustrative decision log with made-up numbers, not real results. Round 2 is the overfitting pattern: training up, held-out flat.* ```text round change train held-out verdict 1 define each vibe in one plain sentence +6 +4 keep 2 add rule: any emoji means wholesome +5 +0 revert 3 set needs_review when two vibes compete +2 +3 keep ``` > **Don't hand it the answer key** — Lance Martin's post warns against pasting the content of failing cases into the prompt while you optimize. If training goes up and the test set stays flat, that's overfitting. **What Anthropic reports** - **78.6% → 90.5%** Customer-support eval accuracy — Measured on held-out tickets. - **≈ 1/5** Cost per ticket — Roughly a fifth of where it started. - **~66% → 88%** The claude-api skill itself — Also reported in the same post. *Anthropic-reported results from Lance Martin's post on claude.dev (Sep 28, 2026). Not our numbers.* ## Three routes through the whole series Four parts, a lot of commands. Here's how they chain together for the three kinds of work you're most likely doing: **Pick your route** - **What are you shipping?** — Whatever the route, write the check down first. - **A UI change (Part 2)** — Checkable brief → /example-skills:frontend-design → screenshots at real widths → /code-review → /simplify. - **A big repo change (Part 3)** — ultracode to investigate → /batch to fan out → /compact between phases → git as the safety net. - **A Claude-powered app (Parts 1 and 4)** — prompt-audit → build-eval → hillclimb, judged on held-out cases. - Then **Did the check pass?** — Tests, a build, screenshots or an eval score, with the output in front of you. - Finally **Ship it with receipts** — Commit, and put the check output in the PR. ## The one rule across all four parts Every command in this series is a bigger hammer than the one before it, and none of them replaces knowing what "done" looks like. Reach for the lightest command that still hands you something you can verify, whether that's a passing test, a screenshot or a held-out score, and only scale up when that stops being enough. The question Thariq Shihipar and Sid Bidasaria ask about workflows works for everything here: does it really need more compute? It's how we run this blog, too. Our pipeline is an eval loop in disguise: the research file is the ground truth, a separate reviewer agent is the grader, and nothing gets committed until the checks have actually run with their output in the transcript. Same idea, smaller stakes than a support bot. 🫡 ## The whole series 1. [Part 1: Audit your prompts with /claude-api prompt-audit](https://bytevancer.com/blog/claude-code-commands-part-1-prompt-audit) 2. [Part 2: /effort, /simplify and /frontend-design](https://bytevancer.com/blog/claude-code-commands-part-2-effort-simplify-frontend-design) 3. [Part 3: ultracode, /batch, /compact and /rewind](https://bytevancer.com/blog/claude-code-commands-part-3-ultracode-batch-compact-rewind) 4. **Part 4: build-eval and hillclimb for your Claude app (this post)** ## Frequently asked questions **What does /claude-api build-eval do?** It builds an eval set for your Claude-powered app. According to Anthropic's claude.dev post, it interviews you, samples inputs such as production data and bug reports, proposes graders, and validates results with confidence intervals. It needs Claude Code v2.1.259 or later. **What does /claude-api hillclimb do?** It iteratively improves your app against an existing eval. It splits the data into training and test sets, proposes a single change per round, and reverts changes that improve training scores without a gain on the test set. It also needs Claude Code v2.1.259 or later. **What makes an LLM eval trustworthy?** Lance Martin's claude.dev post lists four properties: the tasks mirror production, scores rise with stronger models or more effort, there's headroom below 100% even at the frontier, and variance is low. Ambiguous graders and tasks that always fail are signs the eval is miscalibrated. **Which type of grader should I use for an eval?** Anthropic's "Demystifying evals for AI agents" describes three: code-based graders are fast and objective but brittle, model-based graders are flexible but non-deterministic and need calibration, and human graders are the gold standard but slow and expensive. Many evals combine them. **How do I know if hillclimbing is overfitting?** Watch the held-out test set. If training scores go up while the test set stays flat, that's overfitting. hillclimb reverts changes like that, and Anthropic's post also warns against pasting the content of failing cases into the prompt while optimizing. ## Keep reading - [Claude Code Commands, Part 3: ultracode, /batch, /compact and /rewind](https://bytevancer.com/blog/claude-code-commands-part-3-ultracode-batch-compact-rewind) — Part 3 of our Claude Code commands series: ultracode workflows, /batch across worktrees, /compact with focus, and what /rewind can and can't undo. - [Harness vs Loop vs Graph Engineering: Which Layer Is Broken?](https://bytevancer.com/blog/harness-vs-loop-vs-graph-engineering) — Harness vs loop vs graph engineering for AI agents: what each layer does, how they nest, and a symptom table to find which one your agent needs fixed. - [All ByteVancer engineering articles](https://bytevancer.com/blog) ## Want proof your AI feature actually got better? We build Claude-powered cloud applications with evals from day one: real cases, graders you can trust, and changes kept only when held-out scores move. See our [LLM evaluation and tuning for cloud applications](https://bytevancer.com/services/cloud-applications) or [talk to the team](https://bytevancer.com/contact). ## Sources - [Claude Code docs — Skills: bundled skills and /claude-api workflows (checked Oct 3, 2026)](https://code.claude.com/docs/en/skills) - [Lance Martin — Automating eval design and hillclimbing with Claude, claude.dev (Sep 28, 2026)](https://claude.dev/blog/automating-eval-design-and-hillclimbing/) - [Anthropic Engineering — Demystifying evals for AI agents (Jan 9, 2026)](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) - [Claude Code docs — Commands reference (checked Oct 3, 2026)](https://code.claude.com/docs/en/commands) - [Thariq Shihipar & Sid Bidasaria — A harness for every task: dynamic workflows in Claude Code, claude.dev (Jun 2, 2026)](https://claude.dev/blog/a-harness-for-every-task-dynamic-workflows-in-claude-code/) --- *This is the Markdown copy of https://bytevancer.com/blog/claude-code-commands-part-4-build-eval-hillclimb — the page itself has the interactive charts and diagrams.*