研究人员开发了两种新颖的方法，BAL-A 和 BMP-A，用于高效地投毒离线人类反馈强化学习 (RLHF) 管道（如直接偏好优化 (DPO)）中使用的偏好数据集。这些攻击利用了通过翻转偏好标签引起的 DPO 梯度中的参数无关偏移。这些方法将投毒问题转化为结构化二元稀疏近似问题，其中 BAL-A 利用格嵌入，BMP-A 采用二元匹配追踪。在合成数据和斯坦福人类偏好数据集上的实验证明了这些攻击的有效性，展示了数据集几何形状如何影响其成功。 AI
影响 强调了 RLHF 训练数据潜在的漏洞，需要在部署模型时采取强大的数据验证和安全措施。
排序理由 学术论文，详细介绍了 RLHF 管道的新颖攻击方法。
AI 生成摘要 · Google Gemini · 来自 3 个来源。 我们如何撰写摘要 →
arXiv:2604.19117v2 Announce Type: replace Abstract: When a language model agrees with a user's false belief, is it failing to detect the error, or noticing and agreeing anyway? We show the latter. Across twelve open-weight models from five labs, spanning small to frontier scale, …
arXiv:2605.02495v1 Announce Type: cross Abstract: Offline Reinforcement Learning from Human Feedback (RLHF) pipelines such as Direct Preference Optimization (DPO) train on a pre-collected preference dataset, which makes them vulnerable to preference poisoning attack. We study lab…
Offline Reinforcement Learning from Human Feedback (RLHF) pipelines such as Direct Preference Optimization (DPO) train on a pre-collected preference dataset, which makes them vulnerable to preference poisoning attack. We study label flip attacks against log-linear DPO. We first i…