Language models learn to generate facial responses from speech

Researchers have developed a framework to generate appropriate facial responses for a listener in social interactions based on the speaker's words. This approach treats quantized facial gesture elements as additional language tokens for a transformer-based large language model. Initializing the transformer with pre-trained language model weights yielded higher quality responses than training from scratch, demonstrating fluent and semantically relevant generated motion. AI

IMPACT Demonstrates LLMs' potential for multimodal understanding and generation, extending their capabilities beyond text.

RANK_REASON Academic paper detailing a novel framework for LLMs to generate facial responses. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CV →

• Language Models
• Sanjay Subramanian
• Transformer
• VQ-VAE

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →