MuVAP: Multimodal Multiparty Voice Activity Projection for Turn-taking Prediction in the Wild
Researchers have introduced MuVAP, a novel multimodal framework designed for predicting turn-taking in multiparty conversations. This system extends Voice Activity Projection by integrating acoustic predictions with face tracking from a single camera and monaural audio stream, making it suitable for human-robot interaction. To handle the complexity of multiple speakers, MuVAP employs Role-Relative Projection. The framework is validated using the newly created Audio-Visual Conversation Corpus, a 31-hour dataset of unedited conversations, and demonstrates superior performance on turn-taking prediction tasks compared to existing baselines. AI
IMPACT This framework could enhance human-robot interaction by enabling more natural turn-taking in conversations.