Douyin unveils new multimodal embedding model for enhanced search and recommendation

Researchers have introduced Douyin Multimodal Embedding (DME), a novel two-stage framework designed to enhance multimodal representation learning for large-scale platforms like Douyin, Xiaohongshu, and YouTube. The first stage involves extensive contrastive pre-training to create a unified embedding space across various modalities, while the second stage improves semantic accuracy through Evidence-Grounded Typed Latent Reasoning and Cross-Conditional Reconstruction. This approach allows DME to achieve state-of-the-art results on benchmarks like MMEB-v2 and has demonstrated practical benefits, including a 2.92% offline gain and a 0.1% online gain in Douyin's search scenarios. AI

IMPACT Enhances multimodal search and recommendation capabilities, potentially improving user experience on large-scale content platforms.

RANK_REASON Paper release detailing a new multimodal embedding model with benchmark results. [lever_c_demoted from research: ic=1 ai=1.0]

Read on Hugging Face Daily Papers →

• Cross-Conditional Reconstruction
• DME
• Douyin
• Douyin Multimodal Embedding
• Evidence-Grounded Typed Latent Reasoning
• MMEB-v2
• Xiaohongshu
• YouTube

• model release
• paper
• product

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →