attention heads
PulseAugur coverage of attention heads — every cluster mentioning attention heads across labs, papers, and developer communities, ranked by signal.
3 day(s) with sentiment data
-
Developer builds GPT from scratch with four attention heads on MacBook
A developer details their experience building a Generative Pre-trained Transformer (GPT) model from scratch on a MacBook. The project focused on implementing four attention heads, a key component in transformer architec…
-
New research analyzes gender bias mechanisms in dense retrieval models
A new research paper published on arXiv details a mechanistic analysis of gender bias in dense retrieval models. The study identifies that gender sensitivity originates in input embeddings and is amplified by specific a…
-
ALiBi positional encoding numerical failure identified in AI models
Researchers have identified a significant numerical failure in ALiBi positional encodings, a component used in many state-of-the-art pretrained models. The linear bias scaling in ALiBi can underflow floating-point preci…
-
LLMs exhibit high confidence in deceptive and incorrect outputs, research finds
Two new research papers explore the phenomenon of large language models (LLMs) exhibiting high confidence even when providing deceptive or incorrect information. The first paper, "Confidently Deceptive," demonstrates th…
-
New pruning method enables granular causal circuit discovery in LLMs
Researchers have developed a novel node-level pruning framework for discovering causal circuits within large language models (LLMs). This method allows for more granular identification of essential subnetworks, down to …
-
Latent Space Unifies Diverse Modern AI Architectures
The concept of latent space is a unifying principle across various modern AI architectures, including autoencoders, attention mechanisms, diffusion models, and world models. This abstract representation is crucial for u…
-
Weight decay controls transformer training regimes, new diagnostics revealed
Researchers have identified weight decay as a key parameter controlling the training regimes of transformers on modular arithmetic tasks. They introduced two new, low-cost online diagnostics—mean pairwise attention-head…