New CAPE-T2V framework enhances text-to-video generation alignment

Researchers have introduced CAPE-T2V, a novel framework designed to improve text-to-video generation by addressing the mismatch between training captions and inference-time prompts. The system first fine-tunes a prompt enhancer (PE) to generate various target prompts from source captions. Subsequently, it fine-tunes the text-to-video diffusion transformer (DiT) using these enhanced captions, ensuring better alignment between the PE's output and the DiT's training data. This approach leads to improved performance on benchmarks like StoryEval and VBench 2.0, demonstrating a reduced "PE-Caption gap." AI

IMPACT Improves text-to-video generation quality by reducing the gap between training and inference prompts.

RANK_REASON The cluster contains a research paper detailing a new method for text-to-video generation. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CV →

• CAPE-T2V
• LTX 2.3
• StoryEval
• T2V-CompBench
• VBench 2.0
• Wan2.2

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →