Brief

A recent study on unified multimodal models found that Direct Preference Optimization (DPO) struggles to simultaneously improve both image understanding and generation capabilities. The research indicated that generation quality resisted DPO alignment, with one model showing degraded generation performance and another exhibiting near-orthogonal gradients between understanding and generation tasks. This interference is attributed to a significant imbalance in token magnitudes, suggesting discrete VQ tokenization as a potential bottleneck for unified models. AI

Researchers have developed PREFINE, a novel method for adapting pre-trained reinforcement learning policies to incorporate safety constraints without full retraining. This technique leverages trajectory-level preferences, similar to how Direct Preference Optimization (DPO) is used for LLMs, to fine-tune policies for safer behavior. PREFINE has demonstrated a significant reduction in constraint violations and failures, exceeding 60%, while preserving original reward performance. The method offers improved data and computational efficiency compared to traditional offline RL or imitation learning approaches. AI