Gefen optimizer claims 8x memory reduction for LLM training

Gefen is a new optimizer designed as a drop-in replacement for AdamW, aiming to significantly reduce memory usage during model training. The developers claim Gefen can achieve up to an 8x reduction in memory requirements. The project has released its code on GitHub and published a corresponding paper. AI

IMPACT Potentially enables training of larger models or more efficient use of existing hardware for LLM development.

RANK_REASON The item describes a new research paper and associated code release for an optimizer. [lever_c_demoted from research: ic=1 ai=1.0]

Read on r/LocalLLaMA →

• AdamW
• Gefen
• GitHub

• infra
• paper

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →