New FORA technique preserves LLM capabilities during fine-tuning

Researchers have developed a new fine-tuning technique called FORA (Function-space Orthogonal Residual Adaptation) that aims to preserve a large language model's existing capabilities while adapting it to new tasks. Unlike previous methods that focus on weight-space proxies, FORA estimates and protects the activation subspace relevant to a capability. This approach, tested on the Qwen3-1.7B model across tasks like COGS and GSM8K, demonstrated improved preservation over existing methods with minimal trade-off on new task performance. The study suggests that protecting function-space directions is more effective than weight-space projection for maintaining model capabilities. AI

IMPACT This research could lead to more efficient and effective fine-tuning of LLMs, preserving their core abilities while adapting them to specialized tasks.

RANK_REASON The cluster contains a research paper detailing a new fine-tuning technique for large language models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• alphaXiv
• arXiv
• CatalyzeX
• COGS
• DagsHub
• Function-space Orthogonal Residual Adaptation
• Gotit.pub
• GSM8K
• Hugging Face
• Qwen3-1.7B
• ScienceCast

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →