A new research paper proposes that human beliefs about an agent's capabilities significantly influence preferences in Reinforcement Learning from Human Feedback (RLHF). The study introduces a new preference model incorporating these beliefs and a normative theory to bound policy error based on belief mismatches. Empirical evidence from a human study confirms that beliefs affect preferences and can be influenced, suggesting that human labelers often do not assume agent optimality, which can lead to suboptimal RLHF outcomes. AI
IMPACT Highlights a potential avenue for improving AI training by aligning human beliefs with agent capabilities.
RANK_REASON Academic paper on a novel theory within AI research. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
