New SSM adapters outperform LoRA for long-context fine-tuning

Researchers have developed a new parameter-efficient fine-tuning (PEFT) method called Hankel Reduced order Model (HRM) adapters, which utilize state space models (SSMs) for long-context fine-tuning. Unlike traditional PEFT methods that focus on attention mechanisms, HRM adapters are designed to be injected into MLP blocks and leverage the time-invariance of SSMs for efficient computation. In evaluations using Mistral-7B on long-context tasks like LongBench, HRM adapters demonstrated superior performance compared to LoRA variants, achieving significant accuracy and ROUGE-1 score improvements. AI

IMPACT Introduces a novel PEFT method that improves performance on long-context tasks, potentially influencing future model fine-tuning strategies.

RANK_REASON The cluster contains a research paper detailing a new method for fine-tuning language models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• Hankel Reduced order Model
• LongBench
• LoRA
• Mistral-7B
• MLP blocks
• parameter-efficient fine-tuning
• QMSum
• QuALITY
• SSM Adapters
• state space model

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →