Beyond Layer Importance in Layer-wise Sparsity: An Inter-Layer Perturbation-Absorption Perspective
Researchers have developed a new method called absorption-aware correction to improve the efficiency of large language models (LLMs) through pruning. This technique characterizes how different layers in LLMs respond to perturbations, finding that early layers tend to amplify them while later layers absorb them. By incorporating this understanding, the method enhances existing pruning techniques like OWL and AlphaPruning, leading to a 7.13% reduction in perplexity and a 1.02% increase in zero-shot accuracy at 70% sparsity across various model families. AI
IMPACT Enhances LLM efficiency by improving pruning techniques, leading to better performance at higher sparsity levels.