attention heads
PulseAugur coverage of attention heads — every cluster mentioning attention heads across labs, papers, and developer communities, ranked by signal.
1 day(s) with sentiment data
-
New pruning method enables granular causal circuit discovery in LLMs
Researchers have developed a novel node-level pruning framework for discovering causal circuits within large language models (LLMs). This method allows for more granular identification of essential subnetworks, down to …
-
Latent Space Unifies Diverse Modern AI Architectures
The concept of latent space is a unifying principle across various modern AI architectures, including autoencoders, attention mechanisms, diffusion models, and world models. This abstract representation is crucial for u…
-
Weight decay controls transformer training regimes, new diagnostics revealed
Researchers have identified weight decay as a key parameter controlling the training regimes of transformers on modular arithmetic tasks. They introduced two new, low-cost online diagnostics—mean pairwise attention-head…