GPT-2 small
PulseAugur coverage of GPT-2 small — every cluster mentioning GPT-2 small across labs, papers, and developer communities, ranked by signal.
6 day(s) with sentiment data
-
Weight tying in language models: Gradient analysis and performance impact
Researchers explored the implications of weight tying in language models, specifically how tying the input embedding and output projection matrices affects gradient calculations and model performance. They found that de…
-
Mechanistic Tomography framework unifies AI model interpretability methods
Researchers have introduced "Mechanistic Tomography," a framework for interpretability in AI models. This approach unifies various measurement techniques like patching and Hessian-vector products under a shared mathemat…
-
FishBack method improves transformer activation steering using non-Euclidean geometry
Researchers have developed a new method called FishBack to improve activation steering in transformers, a technique for modifying language model behavior without updating parameters. Existing methods are often unstable …
-
RecurrentGPT introduces recurrent modulation for transformer efficiency
Researchers have introduced RecurrentGPT, a novel transformer architecture designed to enhance expressivity and memory efficiency in large language models. This model utilizes recurrent modulation, allowing a shared cor…
-
AI Interpretability Evidence Unreliable for Regulatory Compliance, Study Finds
A new research paper argues that evidence derived from mechanistic interpretability, a method used to understand AI decision-making, is not reliable enough to meet regulatory requirements. The study found that even with…
-
New method uses Koopman operator for model interpretability
Researchers have developed a new method for mechanistic interpretability called "Intrinsic Structure" that uses the Koopman operator to analyze the spectral properties of a model's internal dynamics. This approach aims …
-
New research probes MUON optimizer's convergence and proposes MALT extension
Two new research papers explore the MUON optimization algorithm, a method used in training large language models. The first paper introduces MALT, an extension of MUON that incorporates lightweight diagonal precondition…
-
Researchers explore adaptive depth and cyclic folding for Transformer optimization
Two new research papers explore novel approaches to optimizing Transformer models by dynamically adjusting their depth. The first paper, "Adaptive Depth in Looped Transformers," investigates learned halting gates and tr…
-
GPT-2 Small embedding geometry around "Trump" analyzed
Researchers explored the embedding geometry of the token "Trump" within the GPT-2 Small model's static embedding table. By analyzing nearest neighbors under both discretized and continuous representations of the token's…
-
New framework enhances statistical rigor for AI model interpretability
Researchers have developed Certified Interventional Fidelity (CIF), a new statistical framework designed to rigorously evaluate causal claims in mechanistic interpretability. CIF treats evaluation metrics as causal esti…
-
New CoAx Method Uncovers Self-Repairing Mechanisms in Transformer Circuits
Researchers have developed a new method called Conditional Co-Ablation (CoAx) to better understand how transformer circuits function, particularly when they exhibit self-repairing capabilities. This technique addresses …
-
New VASAE method intrinsically names AI model features with token vocabulary
Researchers have developed a new method called Vocabulary-Aligned Sparse Autoencoder (VASAE) to intrinsically name features learned by sparse autoencoders in transformer models. This approach aligns SAE features with th…
-
New framework certifies interpretability of Sparse Autoencoders in language models
Researchers have developed a new framework to certify the interpretability of Sparse Autoencoders (SAEs) when used with language models. This framework establishes an upper bound on the risk of a language model by using…
-
New IGLU activation function offers improved gradient flow
Researchers have introduced IGLU, a novel parametric activation function for deep neural networks designed to improve gradient flow and optimization stability. Derived from a mixture of GELU gates under a half-normal di…
-
New study finds LMs show some human-like language learning biases
A new research paper explores whether language models (LMs) can offer insights into human language learning by training them on typologically unattested languages. The study, which focused on 12 languages and used GPT-2…
-
New LiFT Framework Uses Linear Programming to Control Transformer Overfitting
Researchers have introduced LiFT, a novel framework for fine-tuning transformer models that utilizes linear programming to control overfitting. This method formulates fine-tuning as a bilevel optimization problem, joint…
-
Muon optimizer needs less orthogonalization than previously thought
Researchers have investigated the optimal level of orthogonalization needed for the Muon optimizer, a technique that enhances neural network training by refining momentum updates. Their study utilized a simplified cubic…
-
New RAG Method Offers Anytime Validity for LLM Swarms
Researchers have developed a sequential extension to Federated Conformal RAG (FC-RAG) called Anytime-FC-RAG, which provides distribution-free coverage for language models at any stopping time. This new method maintains …
-
New Unpack method deciphers transformer component interactions
Researchers have developed a new method called Unpack to analyze the internal workings of transformer models. This technique uses backward recursion to trace how different components, like attention and MLP layers, cont…
-
GPT-2 Small audit finds 'cryptographic keys' feature linked to task failure
Researchers have developed a novel audit pipeline to analyze the internal workings of the GPT-2 Small language model, specifically focusing on its performance on the Indirect Object Identification (IOI) task. The study …