研究人员开发了新的方法来提高用于对齐大型语言模型（LLM）的奖励模型（RM）的可靠性和可解释性。一种方法引入了因果驱动的干预技术，以在推理时减轻 RM 中的各种偏差，显示出对虚假特征的敏感性降低，而没有性能权衡。另一项开发是“reward-lens”库，它将机制可解释性工具应用于 RM，揭示线性归因并不总是能预测因果打补丁的效果。此外，一种称为时间连贯奖励建模（TCRM）的新方法将 RM 视为价值函数，从而能够进行可解释的 token 级奖励轨迹，并提高在基准测试上的性能。 AI
影响 新方法增强了奖励模型的可解释性并减少了偏差，有望实现更可靠的 LLM 对齐和在基准测试上性能的提升。
排序理由 多篇 arXiv 论文介绍了用于改进 LLM 对齐所用奖励模型的新技术和库。
AI 生成摘要 · Google Gemini · 来自 4 个来源。 我们如何撰写摘要 →
arXiv:2604.27495v1 Announce Type: cross Abstract: Reward models (RMs) play a central role in aligning large language models (LLMs) with human preferences. However, RMs are often sensitive to spurious features such as response length. Existing inference-time approaches for mitigat…
Reward models (RMs) play a central role in aligning large language models (LLMs) with human preferences. However, RMs are often sensitive to spurious features such as response length. Existing inference-time approaches for mitigating these biases typically focus exclusively on re…
arXiv:2604.26130v1 Announce Type: cross Abstract: Every RLHF-trained language model is shaped by a reward model, yet the mechanistic interpretability toolkit -- logit lens, direct logit attribution, activation patching, sparse autoencoders -- was built for generative LLMs whose p…
arXiv:2604.22981v1 Announce Type: new Abstract: Reward models in RLHF are trained to score only the final token of a response - a choice that discards rich signal from every intermediate position and produces models whose token-level outputs are noise. We argue this is a missed o…