The claude-api skill introduces two commands—/claude-api build-eval and /claude-api hillclimb—that automate the creation and refinement of evaluations for AI applications. Build-eval guides developers through selecting representative cases, prioritizing production traffic or synthetic data, and generating a grader that scores outputs. It presents a simple page for input review, proposes a cost‑effective grader, and runs a baseline, reporting scores with confidence intervals and providing transcripts for each case. Hillclimb then iteratively improves the application by proposing patches, splitting the evaluation set into train and test, and applying changes only if they improve both sets beyond the evaluation’s noise. It checks for overfitting, reverts regressions, and suggests additional repetitions or cases when improvements are too small to measure. In a customer‑support benchmark, hillclimb reduced token cost from 4.6 to 1 cent per ticket while raising decision accuracy from 74.4% to 98.9%, demonstrating significant efficiency and performance gains. These tools help developers avoid common pitfalls such as evaluation leakage and overfitting, leading to more reliable AI deployments.
Automating eval design and hillclimbing with Claude
The claude-api skill now offers guided commands to build and hillclimb evaluations, helping developers create robust tests and iteratively improve model performance while guarding against overfitting. The build-eval command walks users through selecting production or synthetic cases, grading them, and running baselines with confidence intervals. The hillclimb command then iteratively proposes patches, tests them on train and test splits, and reverts overfitting or regressions, ultimately delivering the best version for the chosen goal. A real‑world example showed cost and accuracy gains on a customer‑support benchmark using Opus and Sonnet models.
Why it matters
Robust evaluation and automated tuning reduce the risk of deploying models that perform well only on crafted tests, ensuring more reliable AI applications in production.