Researchers have developed a new theoretical framework to explain Pervasive Capability Degradation (PCD) in large language models, a phenomenon where fine-tuning for specific tasks degrades general abilities. The theory posits that feature interference, caused by narrow-domain data disrupting the pre-training feature structure, is the root cause. To address this, a novel loss function called AEGIS has been proposed, which directly constrains activation margins to suppress interference while preserving task adaptation. Experiments across multiple model families and tasks show AEGIS outperforms existing methods in maintaining both target task performance and general capabilities. AI
IMPACT This research offers a theoretical understanding and a practical method to mitigate capability loss in LLMs during fine-tuning, potentially improving model robustness and utility.
RANK_REASON Academic paper detailing a new theory and proposed method for LLM capability degradation. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
