Opening article
BigCodeArena: Judging code generations end to end with code executions
BigCodeArena: Judging code generations end to end with code executions — announced via Hugging Face on October 7, 2025. This is why today, we're thrilled to announce BigCodeArena -- the first human-in-the-loop platform for evaluating code generation models through execution. Every code snippet generated by models is automatically executed in isolated sandbox environments. Using automated LLM judges (Claude-3.7-Sonnet) to evaluate code execution results against a GPT-4.1 baseline, we can rapidly benchmark new models. BigCodeArena makes this possible at scale, combining human judgment with real execution feedback to create the most reliable evaluation platform for code generation models.
Model Size
70B
Dense ParametersBenchmark Metric
76%
Reported Accuracy / GainDeployment Platform
Hugging Face
Cloud InfrastructureEcosystem Access
Open Weights
Self-hostable & Fine-tunableThe Core Development
Evaluating the quality of AI-generated code is notoriously difficult. While humans can easily spot whether a piece of code "looks right," determining if it actually works correctly, handles edge cases properly, and produces the intended result requires running and testing it. This is why today, we're thrilled to announce BigCodeArena -- the first human-in-the-loop platform for evaluating code generation models through execution.
Inspired by LMArena for LLMs, we've built a platform that allows anyone to compare code generation models side-by-side, but with a crucial difference: you can actually run the code and see what it produces . Just submit a coding task, watch two different models generate solutions, execute both programs, and vote on which model produced better results. The outcomes are organized into a leaderboard that displays the community's highest-rated models.
The field of code generation has long struggled with reliable evaluation methods. Traditional benchmarks like HumanEval test code against predefined test cases, but these represent only a tiny fraction of real-world programming tasks. Human evaluation platforms exist for general chatbots, but they fall short for code: reading raw source code and mentally simulating its execution is cognitively demanding and error-prone, especially for longer programs or complex UI applications.
Real-World Use Cases & Applications
These results highlight an important insight: aggregate Elo scores don't tell the whole story . Some models remain brittle under specific runtime constraints, and execution environment matters significantly for real-world deployment.
Deployment & Availability
We currently support 10 languages (Python, JavaScript, TypeScript, HTML, C, C++, Java, Go, Rust, and Markdown) and 8 execution environments:
Strategic Perspective & Ecosystem Impact
You ask two AI models to build a responsive photo gallery website. Both generate code that looks syntactically correct. But which one is actually better? Without running the code, it's nearly impossible to tell. One might produce a beautiful, functional grid layout, while the other might have subtle bugs or poor styling that only become apparent when rendered in a browser.
This observation led us to a key insight: execution feedback is essential for humans to judge code quality reliably. That's exactly what BigCodeArena provides.
BigCodeArena extends the Chatbot Arena framework with powerful features specifically designed for code evaluation:
Why it matters
BigCodeArena: Judging code generations end to end with code executions — announced via Hugging Face on October 7, 2025 Reported by Hugging Face on October 7, 2025.