GPT2
PulseAugur coverage of GPT2 — every cluster mentioning GPT2 across labs, papers, and developer communities, ranked by signal.
4 day(s) with sentiment data
-
Hugging Face model repos show inflated sizes due to format duplication
A technical analysis reveals discrepancies between the reported file sizes and actual model weights in several Hugging Face repositories. The author found that summing all files labeled as weights in a repository often …
-
New resk-logits tool protects LLM moderation from jailbreaks
A new tool called resk-logits has been developed to enhance the security of automated listing moderation systems, such as those used by Leboncoin. This GPU-accelerated processor works by applying a penalty to dangerous …
-
TinyML Project Compares Alan V5 and GPT2 Models on ESP32s3
Robbie Rohan details a project involving the Alan V5 model and GPT2, utilizing an ESP32s3 microcontroller for TinyML applications. The initiative focuses on creating small-scale AI models, as outlined in a blog post.
-
New library protects patient booking chatbots from prompt injection
A new library called resk-logits offers a method to protect patient booking chatbots, such as those used by Doctolib, from prompt injection attacks. The library utilizes a GPU-accelerated Aho-Corasick algorithm to penal…
-
Tools Emerge to Verify LLM Outputs and AI Model Lineage
Two projects, Helio and Pedigree for Graph Hacks, tackle the challenge of verifying the correctness of LLM outputs, particularly concerning data lineage and licensing. Helio, a company knowledge graph agent, addresses t…
-
Untied embeddings boost private LLM training accuracy and efficiency
A new study published on arXiv explores the impact of weight tying in decoder-only Large Language Models (LLMs) when fine-tuned using Differentially Private Stochastic Gradient Descent (DP-SGD). The research found that …
-
New method uses GPT2 to model telicity acquisition in children
Researchers have developed a new method called Difference in Surprisal to automatically label telicity in English CHILDES corpora, distinguishing between bounded and unbounded events. This method uses GPT2 token surpris…
-
New law precisely predicts neural network optimization instability
Researchers have identified a precise mathematical law governing the instability of scale-invariant optimization in neural networks. This law reveals a feedback loop between learning-rate schedules and weight decay, med…
-
Transformers exhibit adaptive partial pooling similar to hierarchical regression
A new paper published on arXiv details how transformer models, specifically GPT-2, exhibit adaptive partial pooling. This phenomenon, similar to hierarchical regression, means the model's predictions are influenced by o…
-
New UNIFUSION method adapts autoregressive models to discrete diffusion
Researchers have developed UNIFUSION, a novel method to adapt autoregressive language models into discrete diffusion models. This approach unifies existing diffusion objectives under a single generalized Kullback--Leibl…
-
New library applies GPU-accelerated penalties to LLM logits for enhanced safety
A new open-source library called ReskLogits has been developed to enhance LLM safety by intervening at the logits level, before token sampling occurs. This approach aims to prevent harmful content generation by applying…
-
New optimization method adapts momentum for faster neural network training
Researchers have developed a new optimization method for neural networks that adapts momentum coefficients based on the kinetic energy of each parameter. This approach, inspired by continuous-time dynamics and cubic dam…
-
New research quantifies agreement between data-influence and data-similarity in LLMs
Researchers have quantified the agreement between data-similarity and data-influence measures used to trace LLM outputs back to their training data. Their findings indicate a significant overlap between the two measures…
-
AI Model Time Horizon Projections Show Continued Improvement Trend
A user on Reddit's r/singularity subreddit has used Claude to project forward the time horizon data for AI models, based on historical trends from METR. The current leading model, Mythos, has a task completion time hori…
-
New AI Models Emerge: DiffusionGemma, Qwopus, Hive, and More
Several AI models have been released or highlighted across various platforms. DiffusionGemma 26B-A4B is noted for its parallel text processing capabilities, while Qwopus 3.6 27b-Coder is now available. Additionally, Hiv…
-
PolySAE enhances AI interpretability by modeling feature interactions
Researchers have developed PolySAE, a novel method to enhance sparse autoencoders (SAEs) by modeling feature interactions. Unlike traditional SAEs that assume additive feature combinations, PolySAE incorporates higher-o…
-
New frameworks boost Transformer inference efficiency across devices
Researchers have developed new methods to improve the efficiency of Transformer model inference across multiple devices. One approach, ASTRA, integrates sequence parallelism with mixed-precision attention to reduce inte…
-
Eugene Yan curates essential language modeling papers for study groups
Eugene Yan has compiled a reading list of fundamental language modeling papers, intended to facilitate group study sessions. The list includes seminal works like "Attention Is All You Need," "BERT," and "GPT-3," each ac…