Se-DPO enhances language model training with self-evolving token credit

Researchers have introduced Se-DPO, a novel method for Direct Preference Optimization (DPO) that dynamically adjusts the contribution of individual tokens to the preference signal. Unlike traditional DPO which treats all tokens equally, Se-DPO assigns a 'token credit' based on each token's implicit reward magnitude and confidence. This self-evolving credit mechanism, implemented with a lightweight calibration network, aims to improve alignment as training progresses. Experiments demonstrate significant performance gains, with Se-DPO outperforming standard DPO by up to 12.2 points on benchmarks like Arena-Hard. AI

IMPACT Introduces a more nuanced approach to preference optimization, potentially leading to more aligned and capable language models.

RANK_REASON The cluster contains a research paper detailing a new method for language model training. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• AlpacaEval 2
• Arena-Hard
• arXiv
• Direct Preference Optimization
• Se-DPO

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →