Speech-language models implicitly transcribe spoken words, study finds

A new research paper published on arXiv explores the internal workings of interleaved speech-language models (SLMs). The study reveals that these models, even when not explicitly trained for speech recognition, undergo an implicit transcription phase. In this phase, intermediate layers can decode the text representation of spoken words, with transcriptions appearing as top candidates for a significant portion of the data. Following this, the models predict the next word in the text domain before potentially returning to the speech domain, offering insights into how speech and text modalities interact within SLMs and potentially guiding future optimization. AI

IMPACT Provides insight into the internal mechanisms of speech-language models, potentially guiding future optimization.

RANK_REASON Research paper published on arXiv detailing internal mechanisms of speech-language models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• arXiv
• Hugging Face
• Interleaved Speech Language Models Latently Work In Text
• Logit Lens
• Speech language models
• text LMs

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →