Language Model Neurons Found to Be Sparse, Aiding Interpretability

Researchers have demonstrated that the neurons within a language model's MLP layers exhibit a degree of sparsity comparable to that of Sparse Autoencoders (SAEs). This finding enables the development of a gradient-based pipeline for circuit tracing, allowing for the identification of causally effective neurons. This method has successfully identified circuits of approximately 100 MLP neurons for controlling model behavior on subject-verb agreement tasks and revealed specific neuron sets encoding reasoning steps for multi-hop city-state-capital tasks, advancing automated interpretability without additional training costs. AI

IMPACT Advances automated interpretability of language models by showing MLP neurons are as sparse as SAEs, enabling circuit tracing without extra training.

RANK_REASON This is a research paper detailing a new method for understanding language model interpretability. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• Language Model Circuits
• Lindsey et al.
• Marks et al.
• MLP neurons
• Smolensky
• Sparse Autoencoders (SAEs)

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →