Mural integrates frozen LLMs into image generation via Mixture-of-Transformers

Researchers have developed a new method called Mural that integrates frozen Large Language Models (LLMs) with diffusion-based image generators. This approach utilizes a Mixture-of-Transformers (MoT) architecture to transfer LLM knowledge into text-to-image synthesis without requiring multimodal training data or explicit reasoning supervision. Experiments show Mural achieves strong performance on benchmarks like GenEval and DPG-Bench, and notably exhibits emergent capabilities such as cross-lingual image generation and emoji-directed scene construction. AI

IMPACT This research demonstrates a novel method for leveraging frozen LLM knowledge in image generation, potentially reducing the need for extensive multimodal training data.

RANK_REASON The cluster contains an academic paper detailing a new method for AI model integration. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CV →

• DPG-Bench
• Large Language Models
• Mixture-of-Transformers
• Mural
• WISE

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →