A new benchmark, VoxEntity-EU, has been developed to evaluate how accurately Automatic Speech Recognition (ASR) systems capture specific entities like phone numbers and emails, particularly in German and French. The benchmark revealed that Whisper large-v3-turbo achieved only 34% exact accuracy for these critical data points, while Voxtral Mini 3B performed similarly at 37%. For comparison, transcribing the same text directly yielded 91% accuracy, highlighting a significant gap in ASR performance for precise information extraction, especially over simulated phone lines. AI
IMPACT ASR systems need significant improvement for reliable extraction of critical data like phone numbers and emails, impacting downstream applications.
RANK_REASON The item describes a new benchmark and its results for evaluating ASR systems. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
