Researchers have developed a new framework to evaluate the tool-calling capabilities of large language models (LLMs) when processing voice input, addressing the limitation of existing text-based benchmarks. This framework converts text-based evaluations into audio-based ones by incorporating text-to-speech, speaker variation, and environmental noise, while preserving original annotations. Evaluations on models like Gemini-3.1-Flash-Live and GPT-Realtime-1.5 across converted datasets such as Confetti and When2Call reveal performance variations dependent on the model and task, with Gemini-3.1-Flash-Live scoring highest on Confetti and GPT-Realtime-1.5 on When2Call. The study also noted a performance gap between text and voice inputs, often due to misunderstandings of argument values in speech, and proposed an LLM-as-judge protocol for privacy-preserving evaluation. AI
IMPACT This framework could improve the evaluation of voice-based AI agents, leading to more reliable voice assistants and tools.
RANK_REASON The cluster is about a published academic paper detailing a new framework and evaluation methodology for LLM agents. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
