New pruning method uses Fisher information distances for neural networks

Researchers have introduced a novel parameter pruning technique for neural networks, grounded in differential-geometric distances within model space. This method quantures the minimal distance to a hypersurface where a parameter is zeroed out, using geodesic distance as defined by the Fisher information metric. The approach yields a hierarchy of pruning methods, starting with magnitude pruning and progressing to more advanced, effective schemes. Demonstrated on fully-connected networks and vision transformers across various datasets, this technique consistently outperforms existing methods in accuracy and Matthews correlation coefficient, offering a mathematically-motivated justification for pruning. AI

IMPACT This research offers a novel, mathematically-grounded method for optimizing neural network architectures by improving pruning techniques.

RANK_REASON This is a research paper detailing a new methodology for neural network pruning. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• CIFAR-10
• Fisher Information Distances
• Fisher information metric
• Fully Connected Networks on a Diet With the Mediterranean Matrix Multiplication
• local Fisher information
• magnitude pruning
• Matthews' correlation coefficient
• MNIST database
• Vision Transformers

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →