Introducing the FFASR Leaderboard: Benchmarking ASR in the Real World

Treble Technologies and Hugging Face have launched the Far-Field ASR (FFASR) Leaderboard to evaluate Automatic Speech Recognition models under realistic far-field acoustic conditions. According to the release, the benchmark addresses the persistent frustration where models scoring well on clean, close-microphone datasets degrade substantially when exposed to room reverberation, background noise, and varying microphone distances found in voice interfaces, smart glasses, and humanoid robots. The community-driven leaderboard utilizes Treble's hybrid simulation engine, covering 14 fully furnished simulated rooms ranging from 20 to 470 cubic meters, validated against real-world measurements through a sim-to-real track. Submissions are evaluated across nine conditions, with a primary ranking score determined by four conditions involving target speakers and transient or continuous noise sources across three signal-to-noise ratio tiers. Alongside Word Error Rate, the leaderboard tracks RTFx speed metrics for every submission on an identical NVIDIA L4 GPU to map the Pareto front between accuracy and latency. Early results show that far-field Word Error Rate at low signal-to-noise ratio is consistently several times higher than near-field Word Error Rate on the same speech content. The platform also includes beta moving-source splits to evaluate performance when speakers are in motion, with future plans to add multi-talker scenarios, microphone array support, and echo cancellation to the evaluation roadmap.