Pythia 1B
PulseAugur coverage of Pythia 1B — every cluster mentioning Pythia 1B across labs, papers, and developer communities, ranked by signal.
3 day(s) with sentiment data
-
New distillation method transfers Transformer knowledge to Mamba models
Researchers have developed a new method for distilling knowledge from Transformer models into State Space Models (SSMs) like Mamba. This two-stage approach involves first distilling knowledge into a linearized Attention…
-
Apple researchers distill Transformer knowledge into Mamba architecture
Researchers from Apple Inc. have developed a new method for distilling knowledge from Transformer models into Mamba-like architectures. Their two-stage approach involves first distilling knowledge into a linearized Atte…
-
Pythia language models develop sparse circuits for grammatical agreement
Researchers have investigated the development of grammatical understanding in language models, specifically focusing on subject-verb number agreement. Their study tracked the neural circuits responsible for this grammat…
-
New FTB Graph method maps language circuits in multilingual LLMs
Researchers have developed a method called FTB Graph to analyze the internal workings of multilingual large language models, specifically how they decide which language to generate first. This technique uses Edge Attrib…
-
New RL method slashes LLM pretraining time by 66%
Researchers have developed AC-ODM, a novel method that uses reinforcement learning to optimize the composition of pretraining data for large language models (LLMs). This approach significantly improves sample efficiency…
-
New method validates LLM circuits using ablation tests
Researchers have developed a new method for discovering circuits within large language models by clustering attention head co-activation statistics. This approach, termed "closure-validated circuit discovery," uses caus…
-
Researchers track attention circuit formation in 1B-class language models
A new research paper investigates the emergence of attention circuits in language models, specifically tracking how different types of attention heads form across various model architectures and training datasets. The s…
-
New Sparse Autoencoder Model Enhances LLM Feature Interpretability
Researchers have introduced the Sign-Aware Gated Sparse Autoencoder (SA-GSAE), a novel architecture designed to improve the interpretability of features extracted from Large Language Models. Unlike standard SAEs that en…