Speech representation impacts LLM reasoning, study finds

Researchers have investigated how different speech representations impact the reasoning capabilities of spoken dialogue models. They found that a temporal-granularity mismatch between speech and text tokens can weaken reasoning, as speech tokens are often more temporally redundant. To address this, they introduced a factorized audio language model head and explored various frame rates, identifying 4.17 Hz as an optimal rate for speech question answering with intermediate-layer representation alignment. AI

IMPACT Investigates how speech representation affects LLM reasoning, potentially improving spoken dialogue systems.

RANK_REASON This is a research paper detailing an investigation into speech representation for LLMs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• arXiv
• LLM

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →