FRAME introduces adaptive domain learning for parameter-efficient fine-tuning

Researchers have introduced FRAME, a novel parameter-efficient fine-tuning (PEFT) method that learns the optimal domain for weight updates. Unlike existing methods that fix updates to either the spatial or Fourier domain, FRAME employs a mixture-of-experts approach where each expert learns a unique fractional-Fourier order. This allows the model to dynamically select the most compact domain for each update, reducing interference and improving multi-task composition. FRAME demonstrates improved performance over strong baselines on various benchmarks using Llama-3.1-8B and Qwen2.5-7B models, with learned orders showing interpretable specialization by task and layer. AI

IMPACT FRAME's adaptive domain learning could enhance the efficiency and effectiveness of fine-tuning large language models across diverse tasks.

RANK_REASON The cluster describes a new research paper introducing a novel method for parameter-efficient fine-tuning. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• FlyLoRA
• FourierMoE
• Fractional-Fourier Mixture of Experts
• FRAME
• HMoRA
• Llama-3.1:8b
• LoRA
• parameter-efficient fine-tuning
• qwen2.5:7b

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →