New research highlights RLHF diversity collapse in text-to-image models

Researchers have identified a critical flaw in reinforcement learning from human feedback (RLHF) when applied to flow-matching text-to-image models. While standard policy entropy is often used to maintain diversity in reinforcement learning, this metric proves ineffective for flow models, as it remains constant even when perceptual diversity collapses. This occurs because the fixed noise schedule in flow models leads to a constant entropy, while the mode-seeking nature of policy gradients causes the model to converge to a narrow, high-reward region. To address this, the researchers propose using perceptual entropy, which accurately captures diversity in a perceptual space, and introduce two new strategies: Perceptual Entropy Constraint (PEC) and Perceptual Constraints on Generation Space. Experiments show these methods significantly improve the quality-diversity trade-off, with PEC achieving a score of 0.734 compared to the baseline's 0.366 and enhancing diversity to 0.989 from 0.047. AI

IMPACT Introduces a new metric and methods to improve diversity in text-to-image models trained with RLHF, potentially leading to more varied and higher-quality outputs.

RANK_REASON The cluster contains a research paper detailing a novel approach to a problem in AI model training. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CV →

• Perceptual Constraints on Generation Space
• perceptual entropy
• Perceptual Entropy Constraint
• policy entropy
• reinforcement learning from human feedback
• text-to-image model
• Xiaofeng Tan

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →