Layer pruning harms LLM long-chain reasoning, new study finds

A new arXiv paper reveals that layer pruning, a common technique for optimizing large language models (LLMs), can significantly harm their ability to perform long-chain reasoning. While pruning may not affect general knowledge tasks, it severely degrades performance on complex reasoning benchmarks, even when only a few layers are removed. Standard fine-tuning methods are insufficient to recover this lost reasoning capability, suggesting a need for new pruning strategies that prioritize the preservation of test-time scaling and reasoning robustness. AI

IMPACT Layer pruning techniques may need re-evaluation to avoid degrading critical reasoning abilities in LLMs.

RANK_REASON The cluster contains a research paper published on arXiv detailing findings about LLM capabilities. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• arXiv
• Hugging Face
• Keyu Wang
• Layer Pruning via Fusible Residual Convolutional Block for Deep Neural Networks
• LLMs
• long-chain reasoning
• supervised fine-tuning
• test-time scaling

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →