The Open ASR Leaderboard Adds Its First Global South Language

Voice Arena and Hugging Face have partnered to launch open automated speech recognition evaluation for Hindi and Indian English, marking the addition of the Open ASR Leaderboard's first Global South languages. According to the source, standard leaderboards rely on aggregate word error rates that often conceal performance disparities across different genders, ages, accents, and racial groups. To address this gap, two evaluation sets named Monsoon en-IN and Monsoon hi-IN have been introduced to the leaderboard. Hindi, spoken by over half a billion people, becomes the first Indic language featured on a multilingual tab previously limited to European languages. Each language set includes a public split for self-scoring and a private split withheld to prevent benchmark-specific optimization, with speaker-disjoint splits covering 4,888 speakers and 12 recorded attributes. The Monsoon data was built to vary across nine axes, including geography, age, gender, vocabulary, devices, acoustic environments, speech type, speech rate, and transcript variants. The unscripted dual-channel spontaneous conversations were sourced from contributors using their own everyday handsets and connections across hundreds of districts and multiple states. Recruitment was conducted through the Voice Arena community to reach rural and semi-urban districts rarely covered by traditional speech corpora. By releasing these comprehensive demographic and geographic evaluation sets publicly, the initiative enables developers to analyze and improve model performance across previously underrepresented populations.