English(EN)Inference and learning in sparse autoencoders as natural gradient flow
新研究探讨SAE的局限性并探索分层结构
作者PulseAugur 编辑部·[8 个来源]·
近期研究探讨了稀疏自编码器(SAE)及其相关技术在理解和处理复杂数据方面的局限性和进展。一篇论文研究了SAE在组合泛化方面遇到的困难,认为字典学习是关键瓶颈,而非推理过程本身。另一项研究提出了一种有损压缩文本自编码器,该自编码器在保持语义理解以用于下游任务的同时,实现了具有竞争力的压缩率。此外,关于语音处理的研究表明,使用路由SAE可以区分语言和副语言信息,并且引入了一种名为Tree SAE的新方法来更有效地学习分层特征结构。最后,一个统一的框架将SAE中的推理和学习视为自然梯度流,从而改进了字典恢复和概念检测。
AI
arXiv:2603.28744v2 Announce Type: replace Abstract: The linear representation hypothesis states that neural network activations encode high-level concepts as linear mixtures. However, under superposition, this encoding is a projection from a higher-dimensional concept space into …
arXiv cs.CL
TIER_1English(EN)·Vinko Sabol\v{c}ec, Angelos Katharopoulos, David Grangier·
arXiv:2610.10738v1 Announce Type: new Abstract: Our work explores learning a compressed latent representation of text, at the intersection of data compression and representation learning. We propose an autoencoder architecture that performs residual downscaling and upscaling of h…
arXiv cs.CL
TIER_1English(EN)·Beimnet Bekele Guta, Xiaoyu Yang, Guangzhi Sun, Philip C. Woodland·
arXiv:2610.10865v1 Announce Type: new Abstract: Self-supervised speech encoders contain linguistic and paralinguistic information in a shared, entangled representation space. We combine a TopK sparse autoencoder with route-specific supervision and cross-factor adversaries. Across…
arXiv cs.LG
TIER_1English(EN)·Tue M. Cao, Hoang X. Nhat, Raed Alharbi, Phi Le Nguyen, My T. Thai·
arXiv:2605.07922v3 Announce Type: replace Abstract: Learning hierarchical features in Sparse Autoencoders (SAEs) is essential for capturing the structured nature of real-world data and mitigating issues like feature absorption or splitting. Existing works attempt to identify hier…
arXiv cs.AI
TIER_1English(EN)·Hadi Vafaii, Tejas Rao, David Chanin, Thomas Fel, Jacob L. Yates, Bruno Olshausen, David Klindt, Dileep George, Miguel L\'azaro-Gredilla·
arXiv:2610.07389v1 Announce Type: cross Abstract: Sparse autoencoders are widely used to uncover interpretable features in neural networks, yet reliable recovery remains difficult when features overlap or activate infrequently. These challenges involve both inferring which featur…
arXiv cs.LG
TIER_1English(EN)·Nils Grandien, David Steinmann, Felix Friedrich, Kristian Kersting·
arXiv:2606.22994v2 Announce Type: replace Abstract: Sparse autoencoders (SAEs) have become an important tool for unsupervised concept discovery in large models. To make the resulting feature spaces more interpretable and manageable, recent approaches have begun imposing hierarchi…
We present Empirical Variational Autoencoder, a general generative framework for continuous-valued (i.e., non-vector-quantized) sequences. EVA is based on the evidence lower bound of the Variational Autoencoder (VAE) but learns autoregressive latent priors empirically from traini…
Long-context LLM agents accumulate interaction histories that strain KV-cache memory and attention computation. Although sparse attention reduces these costs, heterogeneous cache representations and workflows hinder integration with existing inference engines, while prior sparse-…