Finer Parameter Steps for Low-Rank PEFT: A Controlled Study with CP Tensor Adapters
Researchers have explored the use of tensorized adapters, specifically canonical polyadic (CP) tensor adapters, as an alternative to traditional low-rank adapters (LoRA) in parameter-efficient fine-tuning (PEFT). By using finer capacity increments, CP adapters store significantly fewer trainable scalars per component compared to LoRA ranks, allowing for more granular control over the parameter budget. While CP adapters train stably and fill the gaps between LoRA ranks, their effectiveness varies by task, with some tasks showing early plateaus and others benefiting from additional components before saturation. AI
IMPACT Provides a more granular approach to fine-tuning models, potentially enabling better performance at lower parameter budgets.