English(EN)Pairwise matrices for sparse autoencoders: single-feature inspection mislabels causal axes
新方法增强稀疏自编码器的可解释性和稳定性
作者PulseAugur 编辑部·[6 个来源]·
研究人员开发了新方法来解决稀疏自编码器（SAE）的局限性，SAE用于解释大型语言模型的内部表示。一篇论文介绍了自适应弹性网络SAE（AEN-SAE），这是一种可微分架构，可在不进行启发式重采样的情况下缓解特征饥饿和收缩偏差。另一项研究提出了一种用于分析SAE特征的成对矩阵协议，揭示了单特征检查可能会错误标记因果轴，并且相干性损失与方向模式有关。此外，另一篇论文提出，结合局部顺序辅助损失（如有限差分符号误差）可以提高自编码器重建精度，超出标准的均方误差。
AI
arXiv:2605.05341v1 Announce Type: new Abstract: Sparse autoencoders (SAEs) are used to disentangle the dense, polysemantic internal representations of large language models (LLMs) into interpretable, monosemantic concepts. However, standard $\ell_1$-regularized SAEs suffer from f…
arXiv cs.LG
TIER_1English(EN)·Harvey Dam, Martin Burtscher, Tripti Agarwal, Ganesh Gopalakrishnan·
arXiv:2504.04202v4 Announce Type: replace Abstract: Mean-squared error is the default objective for training autoencoders, yet compressed reconstructions often depend not only on pointwise accuracy but also on preserving local spatial order. We study whether structural auxiliary …
arXiv cs.AI
TIER_1English(EN)·Ruben Fernandez-Boullon, Pablo Magari\~nos-Docampo, Javier Perez-Robles·
arXiv:2605.06494v1 Announce Type: new Abstract: Sparse autoencoders (SAEs) have become central to mechanistic interpretability, decomposing transformer activations into monosemantic features. Yet existing analyses characterise features almost exclusively through top-activating to…
Sparse autoencoders (SAEs) have become central to mechanistic interpretability, decomposing transformer activations into monosemantic features. Yet existing analyses characterise features almost exclusively through top-activating token lists or decoder weight vectors, leaving the…
arXiv cs.LG
TIER_1English(EN)·Michael A. Riegler, Birk Sebastian Frostelid Torpmann-Hagen·
arXiv:2605.03160v1 Announce Type: new Abstract: The standard sparse-autoencoder (SAE) interpretability protocol labels each feature from its top-activating contexts and validates by single-feature steering. We propose the pairwise matrix protocol, co-varying steering coefficient …
Sparse autoencoders (SAEs) are used to disentangle the dense, polysemantic internal representations of large language models (LLMs) into interpretable, monosemantic concepts. However, standard $\ell_1$-regularized SAEs suffer from feature starvation (dead neurons) and shrinkage b…
