Is it agentic enough? Benchmarking open models on your own tooling

Hugging Face has introduced a new evaluation harness designed to benchmark open models and library revisions based on how effectively coding agents can drive them. Traditional benchmarks typically focus only on the final answer, leaving developers blind to the path an agent took, the work involved, and how library changes affect performance. Using the transformers library as a case study, Hugging Face measured the complete agentic process using open models driven by the pi coding agent across identical hardware via Hugging Face Jobs. The harness evaluates runs across three tiers: bare, clone, and skill variants, assessing metrics like turns, latency, tokens, and errors. For large open models, the framework measures effort since task completion rates saturate near 100 percent. For local models, success rates and match percentages remain relevant metrics for gauging specific capabilities. Testing the transformers repository around commits that introduced a dedicated CLI and skill revealed a distinct trade-off. For large models, the skill commit resulted in less time spent on tasks. Conversely, the clone variant showed a significant increase in token consumption because agents read the newly added CLI implementation and usage examples to learn the interface. Hugging Face notes that while this setup measures a one-off discovery cost, real-world usage amortizes this token cost across multiple sessions as agents learn the interface once and apply it repeatedly.