A new research paper introduces Risk-sensitive Alignment via Dominance (RAD), a novel framework for improving the safety of AI models trained with reinforcement learning from human feedback (RLHF). Unlike traditional methods that rely on expected cost constraints, RAD utilizes stochastic dominance to compare entire cost distributions, offering better control over tail risks and potential catastrophic events. The proposed method integrates Optimal Transport and Sinkhorn iterations for efficient end-to-end optimization and introduces quantile-weighted FSD constraints to universally control a broad class of Spectral Risk Measures, allowing for fine-tuning of the model's risk profile. Empirical results show RAD enhances harmlessness while maintaining helpfulness and demonstrates improved robustness on out-of-distribution evaluations. AI
IMPACT Introduces a novel method for controlling tail risks in AI models, potentially leading to more robust and safer AI systems in critical applications.
RANK_REASON The cluster contains a research paper detailing a new methodology for AI safety. [lever_c_demoted from research: ic=1 ai=1.0]
- Optimal Transport
- reinforcement learning from human feedback
- Risk-sensitive Alignment via Dominance
- stochastic dominance
- Yaswanth Chittepu
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
