GPT-2
PulseAugur coverage of GPT-2 — every cluster mentioning GPT-2 across labs, papers, and developer communities, ranked by signal.
- developed by .openai 100%
- instance of arXiv 90%
- instance of CatalyzeX 90%
- instance of large-language models 90%
- instance of LLM 90%
- instance of Adam 90%
- used by byte-pair encoding 90%
- developed Andrej Karpathy 90%
- developed by DistilGPT-2 90%
- used by transformer 80%
- authored by arXiv 70%
- instance of transformer 70%
24 day(s) with sentiment data
-
Activation-cache compression impacts SAE training, new study finds
Researchers have investigated the impact of activation-cache compression on Sparse Autoencoder (SAE) training. Their findings indicate that the effectiveness of compression varies significantly, with some methods like n…
-
GPT-2 self-training alters word dominance and entropy profiles
Researchers have explored the effects of recursive self-training on language models, specifically fine-tuning GPT-2 on its own output over multiple generations. They observed that as the training temperature increases, …
-
LLMs assessed for causal understanding via sentence reversal and entropy production
Researchers have explored a novel method to assess an LLM's understanding of event order and causality by reversing sentence sequences within texts. This approach, inspired by stochastic thermodynamics, quantifies irrev…
-
Author implements LoRA from scratch for GPT-2, reduces perplexity
The author details their experience implementing LoRA (Low-Rank Adaptation) from scratch for a GPT-2 model. They successfully trained a small fraction of the model's weights in a short time, observing a reduction in per…
-
Transformers generate 3-manifold triangulations using 3 exaFLOPs
Researchers have utilized GPT-2 style transformers to generate triangulations of 3-dimensional manifolds by encoding dehydration signatures. Experiments involved models with up to 200 million parameters and 3 exaFLOPs o…
-
Transformers show in-context control capabilities for dynamical systems
Researchers have explored the capability of Transformer-based neural networks to perform low-level control of unstable, nonlinear dynamical systems using in-context learning. By training a GPT-2 style architecture on tr…
-
FILO framework unifies low-rank adaptation for LLMs, improving efficiency
Researchers have introduced FILO, a novel Fisher-Geometric framework designed to unify and improve low-rank adaptation (LoRA) techniques for large language models. FILO addresses the challenge of singular operators in f…
-
New theory certifies transformer attention with quadratic bounds
Researchers have developed a new conic programming theory for certifying the attention mechanisms in transformer models. This theory addresses the NP-hard nature of certifying attention outputs under input perturbations…
-
Coconut method enables LLMs to reason without explicit step-by-step writing
A new method called Coconut enables language models to reason without explicitly writing down their thought processes. Coconut feeds the model's last hidden state back as the next input, bypassing the need for token-bas…
-
Hierarchos Native update boosts performance on Intel and RDNA3 GPUs
Hierarchos Native, a Rust and Vulkan backend for language model operations, has received a significant update. This update focuses on improving performance and compatibility across various hardware, including Intel HD 5…
-
NVIDIA engineer champions open AI; weight-space programming emerges as new paradigm
Chris Alexiuk, a senior product research engineer at NVIDIA, champions open-source AI, emphasizing the importance of accessible weights and training data for democratizing intelligence. He highlights NVIDIA's Nemotron f…
-
New spectral framework enhances scaling for generative foundation models
Researchers have developed a spectral framework to address challenges in scaling generative foundation models, particularly in stable feature learning and hyperparameter transfer across varying widths and depths. This f…
-
Developer builds 80-line BPE tokenizer in TypeScript
A developer has created an 80-line TypeScript implementation of a Byte Pair Encoding (BPE) tokenizer, a method used by models like OpenAI's GPT-2 and Anthropic's Claude Code. The tokenizer, which is reversible and lossl…
-
Andrej Karpathy details LLM training, from data to 'Swiss cheese' flaws
Andrej Karpathy's comprehensive explanation of Large Language Models (LLMs) like ChatGPT breaks down the entire training process, from data collection and tokenization to the Transformer architecture and next-token pred…
-
AI model training time varies from minutes to months based on size and GPUs
Training AI models on cloud GPUs can range from minutes to months, heavily influenced by model size, dataset size, and the number of GPUs used. A small fine-tuning task might take less than five minutes, while training …
-
New LLM research tackles evaluation, memorization, and training efficiency
Multiple research papers explore advancements in Large Language Model (LLM) evaluation and training. One study proposes soft prompt tuning to improve the agreement of open-source LLM judges with human annotations, showi…
-
ChatGPT's 'thinking' is actually rapid token prediction
ChatGPT and similar large language models do not possess genuine thought or reasoning capabilities. Instead, they function by calculating the probability of the next token in a sequence, one token at a time, through a c…
-
TurnTurk-GPT model predicts Turkish conversation turn-taking
Researchers have developed TurnTurk-GPT, a model fine-tuned from a Turkish GPT-2 to predict turn-taking in conversations. This model demonstrates superior performance compared to multilingual detectors and a zero-shot T…
-
New research bridges compositionality gap in GPT-2 models
A new research paper explores the limitations of transformer language models, specifically GPT-2, in compositional generalization. The study found that while these models can combine known facts correctly in-distributio…
-
Weight pruning can damage AI model circuits without affecting accuracy
A new study published on arXiv investigates the failure modes of weight pruning in Transformer language models, specifically using GPT-2 as a test subject. Researchers found that standard metrics like accuracy and perpl…