Brief

Researchers have developed a new framework called DiPOD to address instability in diffusion policy optimization. Existing methods suffer from a "double-drift" phenomenon where optimization can cause the ELBO to diverge from the true log-likelihood, leading to misaligned policy gradients. DiPOD stabilizes training by combining self-distillation with policy-improving gradient updates, using an on-policy ELBO regularizer. This approach has shown improved stability and higher rewards in both diffusion language model post-training and continuous-control diffusion policies. AI