GPT-2
PulseAugur coverage of GPT-2 — every cluster mentioning GPT-2 across labs, papers, and developer communities, ranked by signal.
- developed by OpenAI 100%
- instance of arXiv 90%
- instance of CatalyzeX 90%
- instance of large-language models 90%
- instance of LLM 90%
- used by byte-pair encoding 90%
- developed Andrej Karpathy 90%
- used by arXiv 70%
- used by alphaXiv 70%
- used by Gotit.pub 70%
- used by transformer 70%
- instance of transformer 70%
13 day(s) with sentiment data
-
New layer-wise curriculum learning method enhances LLM compression efficiency
Researchers have developed a novel layer-wise curriculum learning approach for efficient Large Language Model (LLM) compression. This method facilitates knowledge transfer from larger teacher models to smaller student m…
-
AI doomerism seen as marketing hype by Reddit users
A Reddit user argues that much of the current AI doomerism is exaggerated marketing, particularly from companies like OpenAI, to justify future growth and potential IPOs. The user points to past dramatic predictions abo…
-
AI models classify instruments from sheet music images
Researchers have developed a novel method for classifying musical instruments directly from sheet music images, treating the task as a text classification problem. By converting sheet music into a sequence of musical 'w…
-
Temperon training method achieves SAM quality with reduced cost
Researchers have introduced Temperon, a novel training method designed to achieve the quality of Sharpness-Aware Minimization (SAM) while significantly reducing computational costs. Temperon utilizes a two-phase approac…
-
Geoff Hinton calls ChatGPT an 'alien being'
Geoff Hinton, a prominent AI researcher, has recently described ChatGPT as an "alien being." This statement follows his earlier analogies comparing GPT-2 to a caterpillar and GPT-3 to a butterfly, and his past predictio…
-
New research analyzes Z-loss backward geometry in language models
A new paper analyzes Z-loss, a technique used to stabilize language model training, from a backward-pass perspective. The research introduces a "backward-transport" view that separates the Z-loss source from the archite…
-
French-only BabyLM model reveals tokenizer sensitivity
Researchers have developed MéTRON-FR, a 125M parameter GPT-2 model trained exclusively on French text, achieving notable scores on French-specific benchmarks. When evaluated using a cross-lingual GLUE protocol, the mode…
-
AI legal assistants developed for Nepal to improve access to justice · 2 sources tracked
Two research papers introduce AI-powered legal assistants for Nepal, aiming to improve access to justice. The first, NepKANUN, utilizes a retrieval-augmented generation (RAG) framework with a fine-tuned large language m…
-
New 'Functionalizer' pre-tokenizer boosts vocabulary efficiency for LLMs
Researchers have developed "The Functionalizer," a novel pre-tokenizer framework designed to improve subword tokenization efficiency. This system decomposes orthographic and structural variations into a compositional st…
-
New ISO-LoRA optimizer boosts parameter-efficient adaptation for LLMs
Researchers have introduced ISO-LoRA, a novel optimization technique designed to enhance the efficiency of Low-Rank Adaptation (LoRA) for large language models. Unlike traditional LoRA methods that focus solely on the r…
-
OpenAI withheld GPT-2 release due to misuse concerns
OpenAI announced in 2019 that they would not be releasing their GPT-2 language model due to concerns about potential malicious applications. This decision was made to prevent the misuse of the powerful AI technology for…
-
AI Labs Accused of Using Safety Warnings as Marketing Stunt
Recent warnings from AI researchers at labs like Anthropic and OpenAI about AI safety are being questioned as potentially disingenuous marketing tactics. Critics suggest these warnings, dating back to 2019 with OpenAI's…
-
Developer open-sources low-cost tool to prevent LLM data poisoning
A developer has created and open-sourced a lightweight tool called Beatriz Epistemic Gate to combat data poisoning during the fine-tuning of large language models. This tool acts as a proxy, verifying generated text aga…
-
AI Leaders Call for Development Slowdown Amid Existential Risk Warnings
Anthropic CEO Dario Amodei has publicly advocated for a slowdown in AI model development, citing concerns that advancements are outpacing the ability to understand and control them. This stance was echoed by OpenAI CEO …
-
Browser-based LLM inference engine Three-LLM leverages WebGPU for local execution
Ben Houston has developed Three-LLM, a WebGPU-based inference engine that allows Large Language Models (LLMs) to run locally within a web browser. This project leverages Three.js and its WebGPU capabilities to execute L…
-
New DP-Muon method enhances differentially private optimization
Researchers have developed DP-Muon, a novel method for differentially private optimization that utilizes matrix-orthogonalized momentum. This approach addresses the mean distortion introduced when new Gaussian noise is …
-
Looped GPT-BERT model trades parameters for computation in language modeling
Researchers have developed a novel language model called Looped GPT-BERT, which achieves comparable performance to existing models on linguistic and downstream tasks while using fewer parameters. This is accomplished by…
-
UC Berkeley researchers develop bandit-based pruning for transformers
Researchers from the University of California, Berkeley have developed a novel method for pruning large transformer models, including those used in vision and language tasks. This technique, framed as a damage-aware mul…
-
LLMs adapted for time-series forecasting using parameter-efficient methods · 2 sources tracked
Researchers are exploring parameter-efficient methods to adapt large language models (LLMs) for time-series forecasting tasks. One study projects time-series data directly into the embedding space of a GPT-2 model, achi…
-
Child-like AI models reveal structural alignment boosts grammar learning
Researchers utilized child-like language models, specifically GPT-2 style models, to investigate the impact of different caregiver feedback types on grammar learning. The models were fine-tuned using reinforcement learn…