New LoCA method slashes LLM tuning costs with forward-only approach

Researchers have introduced LoCA, a novel two-stage method for parameter-efficient fine-tuning of large language models. This technique aims to reduce the computational burden by replacing repeated end-to-end backpropagation with a single calibration pass. LoCA achieves this by fitting low-rank maps at each transformer block to correct prediction errors and then using these maps for forward-only tuning. Evaluations on Qwen2.5 models demonstrated that LoCA offers lower cross-entropy and significantly reduces GPU peak memory, CPU memory, and per-pass time compared to LoRA. AI

IMPACT Offers a more efficient method for adapting LLMs, potentially lowering the barrier to entry for fine-tuning.

RANK_REASON Academic paper detailing a new method for LLM tuning. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• arXiv
• LoRA
• Qwen2.5
• SmolLM2-1.7B

• paper
• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →