Transformer layer pruning tests yield divergent results

Researchers have identified that the definition of 'layer equivalence' in transformer models is not a fixed property but depends heavily on the testing methodology. Two distinct tests, 'replacement' and 'interchange', can yield significantly different results regarding which layers are deemed safe for pruning. This divergence is particularly noticeable in large-scale models like Qwen3-8B and Llama-3.1-8B, where the gap between these protocols can change the perceived safety of pruning by several factors, even when using the same evaluation metrics. AI

影响 Highlights that current methods for analyzing transformer layer redundancy for compression are inconsistent, potentially impacting model optimization strategies.

排序理由 The cluster contains an academic paper detailing novel research findings on transformer model analysis. [lever_c_demoted from research: ic=1 ai=1.0]

在 arXiv cs.CL 阅读 →

• Llama-3.1-8B
• Pythia
• Qwen3-8B

• 论文
• 模型发布

AI 生成摘要 · Google Gemini · 来自 1 个来源。 我们如何撰写摘要 →