New method converts AR models to diffusion LLMs with minimal data

Researchers have developed a low-budget method for converting autoregressive (AR) models into diffusion language models (dLLMs), enabling parallel generation without extensive retraining. A study comparing two conversion techniques, in-place and frozen-tower, found that the frozen-tower model significantly outperformed the in-place model on benchmarks like HumanEval, achieving an 11.6x improvement. The frozen-tower approach also retained a higher percentage of the parent model's performance on GSM8K and MMLU-Pro tasks, suggesting it is more effective at preserving knowledge during conversion, especially within a limited training budget. AI

IMPACT This research offers a more efficient pathway for adapting existing autoregressive models to diffusion models, potentially reducing the computational cost and data requirements for developing advanced generative AI.

RANK_REASON Academic paper detailing a new method for converting LLMs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• arXiv
• Context-Tower Conversion
• GSM8K
• HumanEval
• MMLU-Pro
• MoE LLMs

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →