Researchers have developed a Multimodal Voice Activity Projection (MM-VAP) framework designed to improve turn-taking prediction for social robots. This framework extends previous audio-only methods by incorporating synchronized audio-visual inputs and a self-supervised future-projection objective. The system utilizes pretrained audio-visual models adapted for the multimodal turn-taking task, employing an inter-speaker attention stage to model relational dynamics and a semantic consistency loss to regularize the output space. Experiments on the NoXi, NoXi+J, and Haru EDR corpora demonstrated improved performance in predicting turn-taking events, particularly for mediation-oriented human-robot interaction. AI
IMPACT This framework could enable more natural and effective human-robot collaboration in social and mediation settings.
RANK_REASON The cluster contains a research paper detailing a new AI framework for robotics. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 2 sources. How we write summaries →
