Researchers have identified an "intensity undershoot" in language models fine-tuned with Direct Preference Optimization (DPO). When instructed to generate text with a specific emotional intensity, models like Llama-3.1-8B and Qwen3_8B produce outputs that are significantly less intense than requested, with gains of only 0.26 for valence and 0.13 for arousal. This phenomenon appears to stem from the training data, which often lacks extreme emotional examples, limiting the model's ability to learn and replicate high-intensity affect. By diversifying the training data to cover a wider range of emotional targets and increasing the extremity of sampled candidates, the models showed improved performance in generating desired emotional intensities. AI
IMPACT Highlights a limitation in current LLM fine-tuning methods that affects their ability to control emotional expression, suggesting a need for more diverse training data.
RANK_REASON Academic paper detailing a specific finding about LLM behavior. [lever_c_demoted from research: ic=1 ai=1.0]
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- EmoBank
- Fazzi et al.
- Llama-3.1:8b
- Qwen3_8B
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
