XMerge method compresses LLM depth without fine-tuning

Researchers have developed XMerge, a novel post-training method designed to compress the depth of Large Language Models (LLMs) without requiring task-specific labels or end-to-end fine-tuning. This technique identifies transformer layers with minimal impact on model output and reconstructs adjacent layers to maintain performance. XMerge has demonstrated superior results across various Llama and Qwen models, outperforming existing methods in reducing perplexity and maintaining calibration, especially under aggressive layer removal. AI

IMPACT This method could enable more efficient deployment of LLMs by reducing their size without significant performance degradation.

RANK_REASON The cluster contains a research paper detailing a new method for LLM depth compression. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• arXiv
• Hugging Face
• Llama
• Massive Multitask Language Understanding
• Qwen
• XMerge

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →