LLM value preferences shaped by post-training methods, study finds

A new arXiv paper investigates how post-training methods influence the value preferences of large language models (LLMs). Researchers found that different post-training paths, including SFT, DPO, and GRPO, amplify opinion imbalances in corpora to varying degrees. The study also indicates that once preferences are amplified, their removal depends on the specific post-training method and the corpus's opinion mixture. Furthermore, these learned preferences can transfer to student models through distillation, even without explicit opinion questions in the distillation corpus. AI

IMPACT Investigates how training data and methods can embed biases in LLMs, impacting their reliability and fairness.

RANK_REASON Academic paper on LLM training methodology and bias. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• arXiv
• Direct Preference Optimization
• Grpo
• Hugging Face
• supervised fine-tuning

• paper
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →