PREFINE: Preference-Based Implicit Reward and Cost Fine-Tuning for Safety Alignment
Researchers have developed PREFINE, a novel method for adapting pre-trained reinforcement learning policies to incorporate safety constraints without full retraining. This technique leverages trajectory-level preferences, similar to how Direct Preference Optimization (DPO) is used for LLMs, to fine-tune policies for safer behavior. PREFINE has demonstrated a significant reduction in constraint violations and failures, exceeding 60%, while preserving original reward performance. The method offers improved data and computational efficiency compared to traditional offline RL or imitation learning approaches. AI
IMPACT Enhances AI safety by enabling cost-aware behavior adaptation in pre-trained models, improving efficiency and reducing failures.