Residual Context Diffusion Language Models
Researchers have introduced Residual Context Diffusion (RCD), a novel module designed to enhance Diffusion Large Language Models (dLLMs). RCD addresses the inefficiency of current dLLMs by recycling computation from discarded tokens, which retain valuable contextual information. This module converts these discarded representations into contextual residuals and reintroduces them in subsequent denoising steps, improving accuracy by 4-11 percentage points with minimal computational overhead. RCD has shown significant improvements, nearly doubling accuracy on challenging AIME tasks and reducing denoising steps substantially. AI
IMPACT Enhances efficiency and accuracy of diffusion-based LLMs, potentially improving performance on complex reasoning tasks.