New FoRA method slashes fine-tuning parameters while boosting accuracy

Researchers have introduced FoRA, a novel parameter-efficient fine-tuning method that prioritizes reducing the number of trainable parameters by selecting informative layers. This approach, which uses a diagonal Fisher score for layer selection and trains LoRA down-projections on the Stiefel manifold, consistently outperforms existing methods like LoRA and DoRA at half the parameter budget. FoRA also demonstrates competitive accuracy compared to AdaLoRA while using significantly fewer parameters, showing consistent gains across various LLaMA-family, Qwen3, and Gemma backbones. AI

IMPACT This new fine-tuning method could enable more efficient training of large language models, making advanced AI more accessible.

RANK_REASON This is a research paper detailing a new method for parameter-efficient fine-tuning. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• AdaLoRA
• DoRA
• Gemma
• LLaMA
• LoRA
• Qwen3

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →