Pythia-160M
PulseAugur coverage of Pythia-160M — every cluster mentioning Pythia-160M across labs, papers, and developer communities, ranked by signal.
4 day(s) with sentiment data
-
New method deciphers feature flow in foundation models
Researchers have developed a new method to understand how foundation models evolve through fine-tuning and editing, focusing on the internal computations of sparse autoencoders (SAEs). By constructing a transition atlas…
-
AI attention mechanisms: SVD compression accelerates rank collapse in pretrained models
A new research paper explores the foundational role of linear algebra in efficient attention mechanisms within AI models. The study unifies fourteen existing works and introduces an original finding: Singular Value Deco…
-
AI models' token movement aligns with optimal transport theory at final layers
A new research paper explores the movement of token states within transformer models, comparing it to optimal transport theory. The study analyzed Pythia-160M and Pythia-410M models, finding that at the final layer, tok…
-
Token representation flow in LLMs is nonlinear, study finds
Researchers have analyzed the flow of token representations within neural networks, finding that this flow is nonlinear and does not follow its own density. Using discrete Langevin models on Pythia-160M and Pythia-410M,…
-
Daedalus-150M: New hybrid LLM architecture optimized for CPU inference
Researchers have developed Daedalus-150M, a novel language model architecture optimized for CPU inference. Unlike traditional models that are scaled down after design, Daedalus-150M was built with CPU constraints in min…
-
New Daedalus-150M model achieves faster CPU inference with hybrid architecture
Researchers have developed Daedalus-150M, a novel language model optimized for efficient CPU inference. This hybrid model combines sparse attention with short convolutions, allowing two-thirds of its architecture to avo…
-
New research reveals pipeline choices skew AI interpretability scores
A new paper published on arXiv highlights significant variance in autointerpretability scores used for comparing sparse autoencoders (SAEs) in language models. Researchers found that differences in evaluation pipelines,…
-
New research questions stability of low-rank training for LLMs
Researchers have demonstrated that the low-rank subspace assumption used in memory-efficient optimizers like GaLore for training large language models is not as stable as previously believed. Their analysis shows that t…
-
Expander SAEs offer parameter-efficient dictionaries for neural network interpretability
Researchers have introduced Expander Sparse Autoencoders (SAEs), a novel approach to interpret neural network activations by using parameter-efficient dictionaries. This method significantly reduces the number of learne…
-
New research probes Transformer energy use, learned linearity, and training dynamics
Recent research explores the intricacies of Transformer models, focusing on their energy consumption, internal linear properties, and training dynamics. One paper introduces a scaling model to predict energy usage durin…
-
DB-KSVD algorithm offers scalable approach to disentangling high-dimensional embedding spaces
Researchers have introduced DB-KSVD, a novel dictionary learning algorithm designed to disentangle high-dimensional embedding spaces in large transformer models. This method adapts the classic KSVD algorithm to scale ef…
-
AI safety research proposes formal framework for computational substrates
This series of posts explores the concept of 'substrates' in AI, which refers to the computational context layers necessary for implementing AI systems. The authors argue that current AI safety research lacks a clear fr…