The Hugging Face Hub hosts over 8,000 text-to-speech (TTS) models, but evaluation has remained fragmented, unstandardized, and heavily reliant on slow human preference scores like MOS or MUSHRA. To solve this scalability challenge, Hugging Face has introduced the Open TTS Leaderboard, which uses objective metrics to evaluate models on complementary aspects of performance. By relying on objective measures, evaluation times drop from weeks to hours. The default view ranks models by macro-average Word Error Rate on English splits, highlighting leading models such as hexgrad/Kokoro-82M, Supertone/supertonic-3, and fishaudio/s2-pro. The leaderboard also evaluates multilingual performance using character error rates for character-based languages, alongside specific toggles for voice cloning capabilities and speaker similarity metrics. Additionally, a streaming tab ranks models by time-to-first-audio using standardized hardware, identifying models like kyutai/pocket-tts as effective for streaming on both GPU and CPU. A Listen tab further allows community members to compare generated outputs and submit logged-in feedback. Hugging Face plans to open-source the evaluation scripts to allow direct community contributions via GitHub.
Open TTS Leaderboard: Scalable Evaluation for Multilingual Text-to-Speech and Voice Cloning
Hugging Face has launched the Open TTS Leaderboard to address the scaling limitations and fragmentation of traditional human-preference arenas for text-to-speech models. By utilizing objective metrics such as Word Error Rate and speaker similarity, the platform reduces evaluation time from weeks to hours. It provides standardized rankings for multilingual performance, voice cloning, streaming capabilities, and model outputs.
Why it matters
Traditional human-preference arenas cannot scale with the rapid pace of open-source text-to-speech releases, leaving open models underrepresented. Objective metric leaderboards offer a scalable way to evaluate thousands of models fairly.
Topic dossiers Hugging Face