New Ex-Omni Model Integrates 3D Facial Animation with LLMs

Researchers have developed Ex-Omni, an open-source model designed to integrate 3D facial animation generation with omni-modal large language models (OLLMs). This model addresses the challenge of bridging LLMs' discrete reasoning with the continuous dynamics of facial motion by using speech units for temporal structure and hidden speech representations for facial cues. Ex-Omni aims to improve human-computer interaction by enabling OLLMs to produce synchronized speech and 3D facial animations, demonstrating faster generation and better audio-visual synchronization compared to existing cascaded methods. AI

IMPACT Enables more natural human-computer interaction by synchronizing LLM-generated speech with 3D facial animations.

RANK_REASON Research paper detailing a new model for multimodal generation. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• Ex-Omni
• Haoyu Zhang
• InstructS2SF-1200K
• Omni-modal large language models

• paper
• model release
• product

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →