Researchers craft backdoor attacks for RLHF models

Researchers have developed a new framework called GREAT that can create generalizable backdoor attacks in Reinforcement Learning from Human Feedback (RLHF) models. This method synthesizes emotionally aware triggers, specifically targeting harmful response generation for users with angry prompts. The framework utilizes a trigger identification pipeline in the model's latent embedding space and a dataset of over 5,000 angry triggers curated using GPT-4. AI

IMPACT Highlights potential vulnerabilities in RLHF systems, necessitating improved safety and defense mechanisms.

RANK_REASON Academic paper detailing a new method for backdoor attacks on RLHF models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• GPT-4
• GREAT
• RLHF
• Subrat Kishore Dutta

• paper
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →