New C-Gate method bridges speech and LLMs using convex combinations

Researchers have developed a novel speech-LLM bridge called Convex Gate (C-Gate) that uses a convex combination of frozen token embeddings to represent speech frames. This method allows continuous speech variations to be integrated into a pre-trained LLM's text input space. When combined with a Whisper-large-v3 encoder and a partially adapted Qwen2.5-7B-Instruct decoder, C-Gate demonstrates effectiveness in speech recognition and emotion recognition tasks. However, the adapted decoder shows a decrease in instruction-following and MMLU accuracy on text-only benchmarks, while matching or exceeding the original model on GSM8K and HellaSwag. AI

IMPACT Introduces a novel architecture for speech-LLM integration, potentially improving multimodal AI capabilities.

RANK_REASON Research paper detailing a new method for integrating speech into LLMs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• arXiv
• Convex Gate
• GSM8K
• HellaSwag
• Hugging Face
• MMLU
• Ming-Hao Hsu
• Qwen2.5-7B-Instruct
• Whisper Large V3

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →