LLM alignment tax selectively degrades task capabilities, study finds

A new study published on arXiv investigates the "alignment tax" in large language models, specifically examining how fine-tuning with Direct Preference Optimization (DPO) affects model capabilities. The research found that increasing the implicit reward margin during fine-tuning leads to task-selective capability loss, with extraction tasks being largely preserved while router tasks degrade significantly. The study utilized the Qwen3.5-4B model and sparse LoRA techniques, observing that the degradation is primarily due to the suppression of rejected completions rather than a change in chosen reward. AI

IMPACT This research highlights that alignment techniques can have varied impacts on different model capabilities, suggesting a need for more nuanced approaches to fine-tuning.

RANK_REASON Academic paper detailing empirical findings on LLM alignment. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• Direct Preference Optimization
• Hugging Face
• LoRA+
• QLoRA
• Qwen3.5 4B

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →