New dialogue system integrates real-time facial generation with speech

Researchers have developed Moshi-Face, a novel full-duplex spoken dialogue system that integrates facial generation with audio processing. This system utilizes a VQ-VAE to encode facial data into discrete tokens and a Face Transformer to generate these tokens non-autoregressively. The result is a model capable of producing synchronized speech and facial expressions in real-time, maintaining dialogue quality while achieving audiovisual alignment at low latency. AI

IMPACT Enables more natural and expressive human-computer interactions by synchronizing speech with facial movements.

RANK_REASON Research paper detailing a new model release. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• Face Transformer
• Moshi
• Moshi-Face
• VQ-VAE

• model release
• product
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →