New Framework Generates Speech from Facial Images

Researchers have developed a novel Face-to-Speech (F2S) framework capable of generating plausible voices from static facial images, addressing the limitation of text-to-speech (TTS) systems that require reference audio. This system utilizes a lightweight Face Adapter to align facial recognition features with the style space of a frozen StyleTTS 2 model. Evaluations on the LRS3 dataset demonstrated highly natural synthesized speech, consistent voice generation, and a face-to-voice retrieval accuracy above chance. Notably, an English-trained adapter also produced fluent Spanish speech, suggesting the face-to-style mapping is largely language-agnostic. AI

IMPACT This research could enable new applications in historical reconstruction, virtual avatars, and accessibility tools by allowing voice generation from visual data.

RANK_REASON The cluster contains an academic paper detailing a new technical approach. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• arXiv
• Face Adapter
• Hugging Face
• Jose A. Gonzalez-Lopez
• LRS3
• StyleTTS 2
• TED talk
• Zero-Shot Face-to-Speech Synthesis

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →