New pruning method preserves LLM reasoning performance

Researchers have developed a new training-free method called Causal Attribution Pruning (CAP) to reduce the size of large language models while preserving their reasoning capabilities. CAP identifies and prunes less critical attention heads by measuring their causal impact on reasoning tasks. This approach demonstrated significant improvements over existing methods like Wanda, particularly on benchmarks such as ARC-Challenge, and showed promise for models like Llama-3 and Mistral-7B-Instruct at moderate sparsity levels. AI

IMPACT This method could lead to more efficient LLMs, reducing inference costs and making advanced reasoning capabilities more accessible.

RANK_REASON The cluster contains an academic paper detailing a new method for pruning large language models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• ARC challenge
• Causal Attribution Pruning
• GSM8K
• Llama 3
• LLaMA-3-8B-Instruct
• Mistral-7B-Instruct
• multilayer perceptron
• StrategyQA
• Wanda

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →