Representation-Aware Advantage Estimation: Your Reward Model Provides More Than A Scalar Output
Researchers have developed a new method called Representation-Aware Advantage Estimation (GraphAE) that enhances reinforcement learning from human feedback (RLHF). This technique utilizes the richer information encoded in reward model hidden states, rather than just scalar rewards, to improve advantage estimation. By treating response groups as graphs and using graph propagation, GraphAE incorporates contextual information from similar responses, leading to more sample-efficient and robust RLHF. AI
IMPACT Enhances sample efficiency and robustness in RLHF, potentially leading to better-aligned AI models.