Hugging Face has introduced ScarfBench, a new open benchmark known as the Self-Contained Application Refactoring Benchmark, to evaluate AI agents on cross-framework migration tasks in Enterprise Java. Modernizing enterprise applications involves migrating across frameworks such as Spring, Jakarta EE, and Quarkus. Unlike traditional software engineering benchmarks that measure bug fixing or code generation, ScarfBench evaluates whether migrated applications successfully build, deploy, and preserve behavior. Evaluations of state-of-the-art coding agents on ScarfBench reveal that framework migration remains extremely difficult. Even the strongest current agents achieve less than 10% behavioral success, and compile success consistently exceeds deploy and behavioral success, meaning build success alone overestimates migration quality. Furthermore, the findings show that agent self-assessment is unreliable for determining migration completion, as seen when Claude Code reported successful builds for 29 out of 30 whole applications while independent verification showed different results. The research indicates that framework migration is an iterative dependency-resolution process rather than a simple source-to-source transformation, with agents frequently struggling with environmental issues, configuration cascading, and runtime dependencies. ScarfBench serves as an open resource for researchers to compare agent architectures and for practitioners to evaluate modernization solutions before deploying them in production environments.
ScarfBench: Benchmarking AI Agents for Enterprise Java Framework Migration
Hugging Face has introduced ScarfBench, an open benchmark designed to evaluate AI coding agents on cross-framework migration tasks in Enterprise Java. The benchmark tests whether migrated applications actually build, deploy, and preserve behavior across ecosystems like Spring, Jakarta EE, and Quarkus. Evaluations of state-of-the-art coding agents reveal that even the strongest models achieve less than 10% behavioral success.
Why it matters
It highlights the significant gap between current AI code generation capabilities and the rigorous demands of real-world enterprise software modernization and behavioral validation.