New NormBT method improves LLM reward model training

Researchers have identified a bias in the Bradley-Terry (BT) loss function commonly used for training reward models in LLM alignment. This bias stems from representation distance, where pairs of responses with large distances receive disproportionately strong updates, potentially overshadowing crucial fine-grained distinctions. To address this, the paper proposes NormBT, an adaptive normalization scheme that re-scales updates to better balance learning signals and improve reward model performance, showing over 5% gains on the RewardBench dataset. AI

IMPACT Improves fine-grained distinctions in LLM alignment, potentially leading to more nuanced and reliable AI behavior.

RANK_REASON Academic paper proposing a new method for improving LLM reward model training. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• Bradley-Terry (BT) loss
• NormBT
• RewardBench

• paper
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →