Researchers have introduced MuVAP, a novel multimodal framework designed for predicting turn-taking in multiparty conversations. This system extends Voice Activity Projection by integrating acoustic predictions with face tracking from a single camera and monaural audio stream, making it suitable for human-robot interaction. To handle the complexity of multiple speakers, MuVAP employs Role-Relative Projection. The framework is validated using the newly created Audio-Visual Conversation Corpus, a 31-hour dataset of unedited conversations, and demonstrates superior performance on turn-taking prediction tasks compared to existing baselines. AI
IMPACT This framework could enhance human-robot interaction by enabling more natural turn-taking in conversations.
RANK_REASON The cluster describes a new research paper published on arXiv detailing a novel framework and dataset for conversational AI.
- Audio-Visual Conversation Corpus
- MuVAP
- Role-Relative Projection
- SHIFT/HOLD
- Voice Activity Projection
AI-generated summary · Google Gemini · from 2 sources. How we write summaries →