Expander SAEs offer parameter-efficient dictionaries for neural network interpretability

Researchers have introduced Expander Sparse Autoencoders (SAEs), a novel approach to interpret neural network activations by using parameter-efficient dictionaries. This method significantly reduces the number of learned decoder values compared to traditional SAEs, making them more scalable for large models. Experiments on models like Pythia, Qwen2.5-3B, and Llama 3.2 1B demonstrate that Expander SAEs achieve a competitive storage-fidelity tradeoff, using substantially fewer parameters while retaining a high percentage of recovered CE-loss. AI

IMPACT This research could lead to more efficient methods for understanding and debugging large neural networks.

RANK_REASON The cluster describes a new research paper published on arXiv detailing a novel method for mechanistic interpretability in neural networks. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• arXiv
• Expander SAEs
• Hugging Face
• Llama 3.2 1B
• Pythia-160M
• Pythia 70M
• Qwen2.5-3B
• Rodrigo Mendoza-Smith
• Sparse Autoencoders

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →