New SALSA-V model synthesizes synchronized audio from silent video

Researchers have developed SALSA-V, a novel multimodal model designed to generate synchronized, high-fidelity audio from silent video content. The model utilizes a masked diffusion objective for audio-conditioned generation and can produce audio of unconstrained length. By incorporating a shortcut loss, SALSA-V achieves rapid, high-quality audio synthesis in as few as eight sampling steps, potentially enabling near real-time applications. AI

IMPACT Enables high-fidelity, synchronized audio generation from video, potentially impacting content creation and real-time applications.

RANK_REASON The cluster contains a research paper describing a new model and its capabilities. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• arXiv
• Hugging Face
• Luca Lanzendörfer

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →