New UNIFUSION method adapts autoregressive models to discrete diffusion

Researchers have developed UNIFUSION, a novel method to adapt autoregressive language models into discrete diffusion models. This approach unifies existing diffusion objectives under a single generalized Kullback--Leibler objective, allowing for seamless switching between different corruption kernels like mask and uniform noise. Evaluations on GPT2 checkpoints demonstrate that UNIFUSION improves the trade-off between generative perplexity and unigram entropy, outperforming other diffusion models on benchmarks like WinoGrande and SIQA. AI

IMPACT This research could lead to more efficient and versatile text generation models by bridging autoregressive and diffusion architectures.

RANK_REASON The cluster contains a research paper detailing a new method for adapting language models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• Auckland University of Technology
• GPT2
• Neural CTMC
• Siqa
• UNIFUSION
• WinoGrande

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →