How Should LLMs Listen While Speaking? A Study of User-Stream Routing in Full-Duplex Spoken Dialogue
Researchers have explored how large language models can effectively process user input while simultaneously generating spoken responses in full-duplex dialogue systems. They compared two methods: channel fusion, which integrates user input directly into the LLM's input stream, and cross-attention routing, which uses external memory accessed via cross-attention. Channel fusion improved semantic grounding and question-answering accuracy but was susceptible to context corruption during interruptions. Cross-attention routing was more robust to interruptions by preserving the generation context, though it showed lower performance on question answering. AI
IMPACT Investigates architectural choices for LLMs in real-time spoken dialogue, impacting future voice assistant and conversational AI development.