Pythia
PulseAugur coverage of Pythia — every cluster mentioning Pythia across labs, papers, and developer communities, ranked by signal.
6 day(s) with sentiment data
-
Transformer model conversion: Wiring knowledge transfer between sizes
A new research paper explores the transferability of knowledge between different sizes of Transformer models, specifically focusing on converting a 1.4 billion parameter model to a 410 million parameter version within t…
-
Transformer theory extended to include feed-forward networks
Researchers have developed an extended dynamical theory for Transformers that incorporates the feed-forward network (FFN) as a local steering field. This new theory suggests that the tangential component of the FFN is c…
-
New metric predicts neural network performance gains from width scaling
Researchers have introduced the "effective alignment dimension" to better understand how neural network width scaling impacts performance on unseen data. This new metric quantifies the signal-noise geometry of activatio…
-
Sparse Autoencoders Offer Interpretable Insights into LLM Data and Behavior · 4 sources tracked
Researchers are exploring the use of sparse autoencoders (SAEs) as a more cost-effective and interpretable method for analyzing large-scale text corpora and understanding the internal workings of large language models. …
-
LLM linguistic competence drives left-right brain activity prediction asymmetry
Researchers have identified a left-right asymmetry in how large language models (LLMs) predict human brain activity, which emerges as the models develop formal linguistic competence. This asymmetry, observed using fMRI …
-
Pythia system autonomously extracts clinical symptoms without fine-tuning
Researchers have developed Pythia, a novel multi-agent system designed for autonomous clinical symptom detection from notes, eliminating the need for fine-tuning. This system optimizes extraction prompts independently, …
-
New theory suggests infrared organization governs Transformer dynamics in LLMs
A new paper proposes that the emergent behaviors in large language models are governed by a physical mechanism involving the reorganization of time-scale density of states (TDOS). Researchers used Pythia language models…
-
Research: Positional Schemes Shape Transformer Attention Head Algebra
A new research paper explores how positional encoding schemes in transformer models influence the spectral algebra of attention heads. The study found that different positional schemes, such as Rotary Positional Embeddi…
-
New method cuts SLM fine-tuning energy use on embedded GPUs
Researchers have developed an energy-efficient method for fine-tuning small language models (SLMs) on resource-constrained embedded devices. The study characterizes the fine-tuning behavior of BERT and Pythia variants o…
-
New research tracks mentalizing and situation modeling in Transformer language models
A new research paper explores the development of situation modeling and mentalizing capabilities in Transformer language models, specifically the Olmo2 and Pythia suites. The study found that accurate performance on fal…
-
AI models forget learned rules mid-training, research finds
A new research paper introduces the concept of "natural ungrokking," describing how language models can learn a rule during pretraining, only to forget it later without any change in the loss curve. The study found that…
-
Keyless Attention mechanism halves KV cache and boosts transformer efficiency
Researchers have introduced Keyless Attention, a novel attention mechanism for transformers that eliminates the key projection entirely, operating solely on queries and values. This approach results in a Value-Only Cach…
-
New benchmarks tackle privacy risks in large language models
Researchers have developed new methods to evaluate membership inference attacks (MIAs) against large language models (LLMs), particularly focusing on audio and text modalities. The first study introduces a systematic ev…
-
AI transparency debate: 'Open weights' insufficient, requires data and value insight
The article "Open Weights, Closed Minds: What AI Transparency Actually Requires" argues that releasing only model weights, a practice termed "open weights," is insufficient for true AI transparency. While this allows us…
-
LLM function-vector heads split into 'writers' and 'cancellers'
Researchers have identified two distinct populations within function-vector (FV) heads in large language models, challenging the assumption that these heads are a homogeneous group. By employing a sign-preserving criter…
-
New framework predicts side effects of AI model steering
Researchers have developed a new framework to predict side effects of using sparse autoencoders (SAEs) to steer language models. This method analyzes feature statistics before intervention to forecast issues like incons…
-
LLMs Crystallize Factual Knowledge Late in Layers, Study Finds
Researchers have identified a phenomenon called "Late Crystallization" in large language models, where factual knowledge primarily emerges in the final layers rather than gradually across all layers. This finding, obser…
-
Study: Language model circuits vary by architecture
A new study published on arXiv investigates how different language model architectures implement similar task functionalities. Researchers found that the specific circuits responsible for task execution vary significant…
-
New metric predicts language processing costs beyond surprisal
Researchers have introduced a new metric called trajectory extrapolation error to better predict human language processing costs. This metric analyzes the trajectory of hidden states in transformer language models, goin…
-
AI circuit discovery methods may misinterpret structure for function
Researchers have identified a phenomenon called "phantom specialization" in AI models, where variations in input statistics can lead to structurally different circuits that perform the same function. This suggests that …