Adding Benchmaxxer Repellant to the Open ASR Leaderboard
Hugging Face has updated the Open ASR Leaderboard to include private English ASR datasets from Appen Inc. and DataoceanAI. The addition aims to prevent benchmaxxing and test-set contamination while providing a more trustworthy measure of real-world speech recognition performance. The default average WER will continue to be computed on public datasets only, but users can toggle on private datasets to view their impact.
The Core Development
"When a measure becomes a target, it ceases to be a good measure." (Goodhart’s Law)
TLDR : Appen Inc. and DataoceanAI have provided high-quality English ASR datasets covering scripted and conversational speech over multiple accents. To prevent potential risks of benchmaxxing or test-set contamination, we will keep these datasets private for a high-quality measure of performance on multiple tasks.
We’re not updating the average WER at this time : by default, the leaderboard’s Average WER remains computed on public datasets only. You can optionally include the private datasets using the toggle to see their impact 👀
Real-World Use Cases & Applications
Standardization and openness are essential for meaningful benchmarking, but they also make benchmarks more susceptible to benchmark-specific optimization ("benchmaxxing"), where models improve leaderboard performance without corresponding gains in real-world robustness. As models and use cases evolve, the Open ASR Leaderboard will continue incorporating high-quality datasets and new evaluation settings to better reflect real-world performance and improve robustness against benchmark-specific optimization.
The "Rank Δ" column shows how the ordering changes relative to the default macroaverage configuration. Including or excluding public datasets also changes the macroaverage, allowing users to tailor the evaluation to the use cases and data distributions most relevant to their application.
We’re excited to hear the community’s feedback on how the new track and dataset toggling features help users identify the model(s) that best fit their application(s). We’re also looking into evaluations that better reflect real-world noisy conditions, and you can expect some news on that 😉
Strategic Perspective & Ecosystem Impact
Since its launch in September 2023, the Open ASR Leaderboard has been visited over 710K times. We’re blown away by the community’s interest and motivation to keep pushing speech recognition 🗣️
Two words sum up the objectives (but also challenges) in maintaining a benchmark like the Open ASR Leaderboard:
Standardization : models can have different conventions for their usage and outputs, e.g. with/without punctuation and casing. Datasets have the same challenges and can be structured differently. To this end, all test sets have been gathered into a single dataset on the Hub for easy access and previewing. Moreover, to standardize model outputs and dataset transcripts, we use a normalizer that (among other things) removes punctuation and casing, and maps to American spelling. It is based on the normalizer of Whisper .
Why it matters
Benchmark-specific optimization can lead to inflated leaderboard scores without improving real-world model robustness. Incorporating private evaluation datasets helps maintain the integrity and trustworthiness of community benchmarks.