Brief

Researchers have introduced Discrete Stochastic Localization (DSL), a new continuous-state framework for non-autoregressive generation. This method aims to improve upon existing discrete diffusion models by offering a more flexible representation that is invariant to signal-to-noise ratio. Fine-tuning a pre-trained model with DSL has shown significant improvements in distributional faithfulness on the OpenWebText dataset, even supporting faster sampling with fewer steps. AI

Researchers are exploring new methods to improve the efficiency and scalability of diffusion language models (DLMs) for generating long sequences of text. One approach, Block Approximate Sparse Attention (BA-Att), accelerates attention computation by downsampling the attention space, achieving significant speedups while maintaining near full-attention performance. Another development, Dynamic Chunking Diffusion Models (DCDM), replaces fixed positional blocks with content-defined semantic chunks to better capture sequence structure. Additionally, advancements in continuous diffusion models, like RePlaid, demonstrate competitive performance against discrete DLMs, suggesting they are a viable and scalable alternative. AI