New research advances diffusion language models with efficiency and performance gains · 10 sources tracked
ByPulseAugur Editorial·[54 sources]·
Researchers are developing new methods to improve diffusion language models (DLMs), focusing on efficiency and performance. "Clock Diffusion" introduces semi-autoregressive continuous DLMs with position-dependent noise schedules, achieving state-of-the-art bounds on OpenWebText and outperforming baselines on GSM8K. Another approach, ITC-MoE, uses importance-guided token-aware compression to reduce the computational and storage costs of Mixture-of-Experts DLMs while maintaining generation quality. Other advancements include Hierarchical Continuous Diffusion Language Models (HC-DLM) that couple discrete token generation with continuous latent trajectories, Self-Repulsive Sampling (SR) to diversify model outputs, and Fork-dLLM which hybridizes parallel and autoregressive generation for efficiency. Additionally, Stepwise Risk-Sensitive GRPO enhances reasoning and diversity in DLMs, while Diffusion Layer Integrated Gradients (DLIG) provides a method for analyzing DLM generation dynamics. Finally, research is exploring feature-selective model collapse in diffusion models and fine-tuning techniques like DOHF and GoldiMask to improve performance and control.
AI
IMPACT
These advancements in diffusion language models could lead to more efficient and capable AI systems for tasks requiring complex reasoning and generation.
RANK_REASON
Multiple research papers published on arXiv detailing novel methods and improvements for diffusion language models.
arXiv:2610.12474v1 Announce Type: cross Abstract: Zero-shot text-to-speech (TTS) can clone a voice from seconds of audio. Yet the quality comes from billion-parameter diffusion transformers whose size and multi-step sampling put them out of reach of the on-device, assistive, and …
arXiv:2610.13337v1 Announce Type: cross Abstract: Attention sinks are tokens or positions that consistently receive disproportionate attention mass, often beyond what their semantic content would suggest. Despite extensive study in large language models, their behavior and functi…
arXiv:2602.22871v2 Announce Type: replace-cross Abstract: Reasoning with large language models often benefits from generating multiple chains-of-thought, but existing aggregation strategies are typically trajectory-level (e.g., selecting the best trace or voting on the final answ…
arXiv:2610.10990v1 Announce Type: cross Abstract: Unified multimodal diffusion large language models (dLLMs) offer a single architecture for both image generation and multimodal understanding, but their iterative decoding requires tens to hundreds of forward passes. Existing few-…
arXiv cs.LG
TIER_1English(EN)·Gaurav Patel, Jun Fang, Greg Ver Steeg, Qiang Qiu, Sravan Sripada·
arXiv:2610.10859v1 Announce Type: cross Abstract: Text-to-image diffusion models are increasingly distilled into few-step variants and being deployed to enable fast inference. However, their ability to generate harmful or undesired content poses significant safety risks. Data-dri…
arXiv:2610.09145v1 Announce Type: cross Abstract: We revisit a standard accepted practice in the continuous diffusion language model literature of fixing conditioning prompt tokens clean during training. We make a very simple modification: also noise the conditioning prompt token…
arXiv:2610.08833v1 Announce Type: new Abstract: Masked diffusion language models (MDLMs) decode by repeatedly committing tokens to masked positions, but these commitments are usually irreversible. A token chosen under sparse, partial context is kept fixed, even when later context…
arXiv cs.LG
TIER_1English(EN)·Pawe{\l} Skier\'s, Ma{\l}gorzata Grzanka, Wojciech Masarczyk, Jan-Willem van de Meent, Kamil Deja·
arXiv:2610.09876v1 Announce Type: cross Abstract: Diffusion models generate realistic images but often fail on visual reasoning tasks, such as filling in a Sudoku or drawing the path through a maze. When a discrete symbolic representation is available, recursive methods such as t…
arXiv:2605.10971v2 Announce Type: replace-cross Abstract: Discrete diffusion language models (DLMs) generate text by iteratively denoising all positions in parallel, offering an alternative to autoregressive models. Controlled generation methods for DLMs, imported from autoregres…
Text-to-image diffusion models are increasingly distilled into few-step variants and being deployed to enable fast inference. However, their ability to generate harmful or undesired content poses significant safety risks. Data-driven unlearning methods suppress targeted generatio…
arXiv cs.LG
TIER_1English(EN)·Yiming Qin, Ke Wang, Amel Abdelraheem, Adam Hazimeh, Pascal Frossard·
arXiv:2610.08108v1 Announce Type: new Abstract: Diffusion language models (dLLMs) have emerged as a promising alternative to autoregressive (AR) language models, offering flexible token-update orders and parallel decoding. Recent dLLMs are often initialized from pretrained AR mod…
arXiv:2610.08738v1 Announce Type: new Abstract: Diffusion Language Models (DLMs) hold the promise of order-agnostic, parallel text generation. Recently, continuous diffusion and flow matching models have seen substantial gains, driven by carefully crafted token representations an…
arXiv:2610.07028v1 Announce Type: new Abstract: Diffusion-based world models can generate and predict trajectories in high-dimensional dynamical systems, but predictive accuracy does not imply that their latent coordinates recover the underlying state variables or causal interact…
arXiv:2610.07121v1 Announce Type: new Abstract: Diffusion large language models dLLMs) have emerged as a promising alternative to autoregressive language models through bidirectional diffusion-based token generation. However, their growing model sizes and high inference costs mak…
Diffusion language models (dLLMs) have emerged as a promising alternative to autoregressive (AR) language models, offering flexible token-update orders and parallel decoding. Recent dLLMs are often initialized from pretrained AR models before diffusion conversion in order to inhe…
arXiv cs.CL
TIER_1English(EN)·Seo Hyun Kim, Sunwoo Hong, Younwoo Choi, Chen-Hao Chao, Se-Young Yun, Rahul G. Krishnan·
arXiv:2610.03665v1 Announce Type: cross Abstract: Masked diffusion language models (dLMs) offer a promising parallel alternative to autoregressive models for complex reasoning. However, they face a distinct credit-assignment challenge, since a few commitments during denoising sha…
arXiv cs.AI
TIER_1Français(FR)·Zhihan Yang, Wei Guo, Jean-Marie Lemercier, Simon Welker, Yonggan Fu, Mohammad Mahdi Kamani, Sajad Norouzi, Julius Berner, Tomas Geffner, Karsten Kreis, Yongxin Chen, Molei Tao, John Thickstun, Pavlo Molchanov, Ante Juki\'c, Arash Vahdat, Morteza Mardani·
arXiv:2610.02665v1 Announce Type: cross Abstract: Despite the success of discrete diffusion language models (dLMs) for fast parallel decoding, their non-smooth, high-dimensional space hinders trajectory steering for reasoning and inference acceleration. To overcome this, we prese…
Masked diffusion language models (dLLMs) generate text by iteratively denoising masked positions, re-predicting each token multiple times before it is committed. An autoregressive decoder exposes an answer's distribution once, at the step that commits it; a dLLM exposes it at eve…
Diffusion large language models dLLMs) have emerged as a promising alternative to autoregressive language models through bidirectional diffusion-based token generation. However, their growing model sizes and high inference costs make efficient deployment challenging: full-sequenc…
Multimodal Diffusion Transformers (MM-DiTs) jointly process visual and textual representations throughout generation. These models repeatedly update the text tokens through multimodal attention, forming dynamic contextual tokens whose function is not well understood. In this work…
We introduce a post-training method for diffusion language models (DLMs) that minimizes Maximum Mean Discrepancy (MMD) between generated and reference distributions in the feature space of a frozen pretrained DLM. To estimate MMD, we retain contextual features at individual token…
arXiv cs.LG
TIER_1English(EN)·Yair Schiff, Omer Belhasin, Roy Uziel, Matan Rusanovsky, Ran Zilberstein, Marianne Arriola, Gilad Turok, Guanghan Wang, Volodymyr Kuleshov, Michael Elad·
arXiv:2610.00894v1 Announce Type: new Abstract: Recent works on continuous diffusion for discrete data have demonstrated performance on par with comparable discrete diffusion models. However, these continuous counterparts lack key features that are essential to practical use as l…
Diffusion language models (DLMs) enable fast generation by predicting multiple tokens in parallel, but their practical adoption remains limited by a persistent quality gap relative to comparably sized autoregressive (AR) models. We attribute this gap to a computation-difficulty m…
arXiv cs.AI
TIER_1English(EN)·Zhengyi Guo, Jiayuan Sheng, Wenpin Tang, David D. Yao·
arXiv:2609.31882v2 Announce Type: replace-cross Abstract: Reward-based diffusion fine-tuning faces practical challenges when desirable outcomes are rare or conditioning corrections are costly to estimate. In this work, we propose Diffusion Online $h$-guidance Fine-tuning (DOHF), …
arXiv:2610.01296v1 Announce Type: new Abstract: Mixture-of-Experts (MoE) Diffusion Language Models (DLMs) offer flexible parallel decoding and increased model capacity, but their large number of expert parameters incurs substantial computation and storage costs. Existing low-rank…
arXiv:2610.00661v1 Announce Type: cross Abstract: Diffusion large language models (dLLMs) generate text by denoising a sequence or successive blocks, allowing several tokens to be revealed in parallel. Reinforcement learning with verifiable rewards (RLVR) reuses terminal feedback…
arXiv:2610.01177v1 Announce Type: cross Abstract: This work presents Diffusion Layer Integrated Gradients (DLIG), a token attribution method for diffusion language models (DLMs) that extends Integrated Gradients (IG~\cite{sundararajan2017axiomatic}) to arbitrary layers and denois…
arXiv cs.AI
TIER_1English(EN)·Hanna Malet, Gabriel Turinici·
arXiv:2610.01318v1 Announce Type: cross Abstract: Model collapse arises when generative models are trained on synthetic data produced by earlier models. The phenomenon has attracted considerable attention because of its societal and technical implications. However, previous studi…
arXiv:2610.02193v1 Announce Type: cross Abstract: Discrete diffusion language models offer a compelling alternative to autoregressive generation for tasks demanding bidirectional reasoning and global constraint satisfaction. Yet they share a structural bottleneck: when decoding i…
arXiv cs.LG
TIER_1English(EN)·Michael Helcig, Martin Jaggi·
arXiv:2609.39560v1 Announce Type: new Abstract: Sampling several responses and voting over their answers can improve a language model's accuracy, but repeated answers limit the benefit of additional samples. Raising temperature increases diversity at a potential cost to per-sampl…
arXiv cs.LG
TIER_1English(EN)·Stipe Frkovi\'c, Metod Jazbec, Christian A. Naesseth·
arXiv:2609.39859v1 Announce Type: new Abstract: Masked diffusion language models (dLLMs) have shown strong potential for faster inference through parallel token generation when combined with confidence-based samplers. However, recent work has shown that such methods can defer unm…
Masked diffusion language models (dLMs) offer a promising parallel alternative to autoregressive models for complex reasoning. However, they face a distinct credit-assignment challenge, since a few commitments during denoising sharply reduce the uncertainty over the remaining mas…
arXiv:2609.38385v1 Announce Type: new Abstract: Supervised fine-tuning of discrete diffusion language models masks some response tokens and trains the model to recover their original values from the visible context. The masking pattern therefore determines both the context availa…
arXiv:2609.38364v1 Announce Type: cross Abstract: Discrete diffusion models and flow matching have emerged as powerful frameworks for generative modeling over discrete state spaces, yet efficient few-step generation remains a fundamental challenge. In this work, we introduce the …
arXiv cs.AI
TIER_1English(EN)·Jiacheng Qiu, Christopher E. Mower, Jan Peters, Haitham Bou-Ammar, Matthieu Zimmer·
arXiv:2609.38536v1 Announce Type: cross Abstract: Diffusion-based large language models (dLLMs) promise to break the sequential latency bottleneck of autoregressive agents through parallel decoding, but recent evaluations show this efficiency does not transfer to embodied agentic…
arXiv:2609.39441v1 Announce Type: cross Abstract: Online reinforcement learning has been extended to flow matching for diffusion model (DM) image generation. However, this paradigm faces three limitations: (1) Window selection. Existing methods manually set the stochastic differe…
arXiv:2609.39625v1 Announce Type: cross Abstract: Sparse autoencoders (SAEs) reveal visual structure in diffusion transformers (DiTs), but interpreting a feature does not establish whether it can be used to control generation. We introduce D-Scope (Diffusion Scope), a framework t…
arXiv cs.CL
TIER_1English(EN)·Paul Le Van Kiem, Dario Shariatian, Umut Simsekli, Alain Durmus·
arXiv:2609.40235v1 Announce Type: cross Abstract: Continuous diffusion language models generate all tokens in parallel, yet high-quality generation can still require hundreds of network evaluations (NFEs). We study how distributional distillation can reduce this cost by exploitin…
arXiv cs.AI
TIER_1English(EN)·Donghyun Kim, Taehyuk Lee, Jinyeong Kim, Youngmin Oh, Dohyeong Kim, Jaehyuk Ryu, Sangwoo Hong·
arXiv:2609.25166v2 Announce Type: replace-cross Abstract: Diffusion data-point unlearning is typically evaluated immediately after each deletion, even though subsequent requests may repeatedly update the same model. We identify sequential reappearance, a failure mode in which an …
Discrete diffusion language models offer a compelling alternative to autoregressive generation for tasks demanding bidirectional reasoning and global constraint satisfaction. Yet they share a structural bottleneck: when decoding in parallel, each token is sampled independently fr…
arXiv cs.LG
TIER_1English(EN)·Nikita Gushchin, Dmitry Baranchuk, Alexander Korotin·
arXiv:2609.38066v1 Announce Type: new Abstract: Discrete diffusion language models can generate multiple tokens in parallel, but reducing the number of denoising steps can lead to inconsistent predictions. Standard cross-entropy training fits conditional token marginals, whereas …
arXiv cs.AI
TIER_1English(EN)·Manuel Madeira, Amitis Shidani, Alice Bizeul, Victor Turrisi, Louis B\'ethune, Bhavika Devnani, Dan Busbridge, Pierre Ablin, Jo\~ao Monteiro·
arXiv:2609.37974v1 Announce Type: cross Abstract: Masked diffusion models (MDMs) generate text by unmasking several tokens per step, but they are trained and sampled under different conditions. The model is trained on randomly masked sequences, whereas inference follows a traject…
arXiv:2602.06155v2 Announce Type: replace Abstract: Diffusion models generate samples through a sequence of learned denoising steps, and recent work has studied how semantic structure appears along this sampling process. We study this question in deterministic samplers by measuri…
arXiv:2609.36452v1 Announce Type: cross Abstract: Masked diffusion language models (MDLMs) can generate text efficiently by predicting multiple masked tokens in parallel, but predictions from the same forward pass are not necessarily reliable when committed together. We study whe…
arXiv:2609.37391v1 Announce Type: new Abstract: Diffusion language models (DLMs) enable parallel generation by predicting and committing multiple tokens at each denoising step, yet they can generate individually plausible but mutually inconsistent tokens. Recent work shows that \…
High-compression tokenizers are essential for scaling latent image generative models. However, aggressive compression creates a fundamental tradeoff between reconstruction fidelity and generation efficiency: high compression image encoder always increases the learning difficulty …
Masked diffusion language models (MDMs) admit flexible generation orders, making the unmasking strategy an inference decision. Existing methods vary in how they prioritize positions, control parallelism, restrict selection regions, revise predictions, or plan future denoising, ye…
arXiv:2610.07337v1 Announce Type: new Abstract: Removing copyrighted, unsafe, or user-specified concepts from a deployed text-to-image diffusion model is now a practical requirement. Weight-editing methods can suppress fixed targets, but they require per-target retraining and mod…
arXiv:2610.01989v1 Announce Type: new Abstract: Concept erasure removes copyright-protected, privacy-sensitive, or otherwise undesirable concepts from pretrained text-to-image diffusion models to support content governance and compliance. As erasure requests arrive over time, mod…
arXiv:2610.01969v1 Announce Type: new Abstract: Concept erasure aims to remove a target concept, such as a copyrighted style, a recognizable character, or unsafe content, from a pretrained text-to-image diffusion model while preserving its ability to generate other content. Exist…
arXiv:2610.01723v1 Announce Type: new Abstract: Text-to-image diffusion models have achieved remarkable progress in image synthesis, yet can exhibit memorization by closely reproducing individual training examples. Effective mitigation must preserve useful prompt information to g…
arXiv:2610.01496v1 Announce Type: new Abstract: Recent work on language-model adaptation has shown that single models can obtain informative training signals by evaluating their behavior in demonstrationor feedback-augmented contexts, with the help of a teacher network, which is …
arXiv:2609.39222v1 Announce Type: new Abstract: High-compression tokenizers are essential for scaling latent image generative models. However, aggressive compression creates a fundamental tradeoff between reconstruction fidelity and generation efficiency: high compression image e…
arXiv cs.CV
TIER_1English(EN)·Arian Komaei Koma, Seyed Amir Kasaei, Aida Aryafar, Matin Ghiasi, Ali Aghayari, Amirhossein Souri, Mohammad Mosayyebi, AmirMahdi Sadeghzadeh, Mohammad Hossein Rohban·
arXiv:2609.37537v1 Announce Type: new Abstract: Machine unlearning has emerged as a critical post-hoc safety measure to erase sensitive concepts from Text-to-Image (T2I) models without prohibitive retraining. However, we reveal that current state-of-the-art (SOTA) approaches are …
