Researchers have developed Alignment-Free Text-Audiobox (Text-AB), a novel framework for voice dubbing and dialogue synthesis. This system utilizes a Diffusion Transformer with a flow-matching objective and operates within a latent diffusion framework using DAC-VAE features for high-fidelity audio compression. A key innovation is its alignment-free approach, which learns text-speech alignment through cross-attention, eliminating the need for explicit alignment or duration prediction. The system was pre-trained on a massive dataset of 480,000 hours of speech, resulting in a 3-billion parameter model capable of high-quality voice dubbing and full-duplex dialogue synthesis, outperforming previous systems in naturalness and expressivity. AI
IMPACT This new framework could significantly improve the quality and efficiency of voice dubbing and dialogue synthesis applications.
RANK_REASON The cluster describes a new research paper detailing a novel framework for text-to-audio synthesis.
Read on Hugging Face Daily Papers →
AI-generated summary · Google Gemini · from 2 sources. How we write summaries →
