StreamTalk framework generates real-time co-speech gestures with improved accuracy

Researchers have developed StreamTalk, a novel framework for generating realistic co-speech gestures in real-time. Unlike previous open-loop methods that suffer from accumulated drift over long sequences, StreamTalk employs a closed-loop system with a generate-retrieve-refine cycle. This approach uses a key pose as an anchor to limit drift and improve trajectory accuracy. The framework also incorporates techniques like Stochastic Anchor Masking and a part-aware DiT to enhance motion recovery and reduce interference between different motion streams, achieving state-of-the-art results on the BEAT2 dataset. AI

IMPACT This research could lead to more natural and engaging virtual avatars and human-computer interactions.

RANK_REASON The cluster contains an academic paper detailing a new method for gesture generation. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CV →

• BEAT2
• StreamTalk

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →