FineWeb-Edu
PulseAugur coverage of FineWeb-Edu — every cluster mentioning FineWeb-Edu across labs, papers, and developer communities, ranked by signal.
7 day(s) with sentiment data
-
User trains 1.1B LLM from scratch for $200, shares code and model
A user has successfully trained a 1.1 billion parameter large language model from scratch for approximately $200. The model, named 'gemmeh', was pre-trained on 20 billion tokens from the fineweb-edu dataset and then fin…
-
New trust-region framework enhances adaptive moment estimation in optimization
Researchers have introduced a novel trust-region framework designed to analyze the behavior of adaptive moment estimation methods in stochastic gradient optimization. This framework constrains the magnitude of update st…
-
New CuraWeb corpus boosts LLM performance with optimized data curation
Researchers have developed CuraWeb, a new 2 trillion token English corpus designed to improve the pretraining data for large language models. Unlike previous methods that focused on singular optimization objectives, Cur…
-
Möbius RoPE enhances in-context retrieval reliability in language models
Researchers have developed a new positional encoding technique called Möbius RoPE, which utilizes anti-periodic boundary conditions to improve in-context retrieval reliability in language models. This method, applied to…
-
User trains 0.5M parameter model Silia-v2 on 1B tokens
A user has trained a small language model with 0.5 million parameters on 1 billion tokens from the FineWeb-Edu dataset. This model, named Silia-v2, is an iteration on a previous research paper and incorporates architect…
-
Researchers analyze Transformer attention mechanisms and feed-forward networks · 2 sources tracked
Two new research papers explore the fundamental components of Transformer models, specifically focusing on the role of attention mechanisms versus feed-forward networks. The first paper, "A Controlled Study of Attention…
-
Depth-Recurrent Transformers Show Per-Token Fixed-Point Convergence
Researchers have investigated the internal computations of depth-recurrent transformers, specifically how each token's state evolves over multiple processing loops. They found that the recurrent state converges to a fix…
-
Small language model trained on single GPU detailed in new study
Researchers have detailed a method for training a small language model, L20-Edu-135M, using significantly fewer computational resources, specifically on a single NVIDIA L20 GPU. The study focused on data efficiency, uti…
-
New pretraining method enhances LLM safety with integrated reflection
Researchers have introduced a new method called Safety Reflection Pretraining, designed to enhance the safety alignment of large language models (LLMs) during the pretraining phase. This approach goes beyond simply filt…
-
EverydayGPT uses confidence gating to cut RAG latency by 120x
Researchers have developed EverydayGPT, a conversational question-answering system that uses a Confidence-Gated Routing (CGR) mechanism to improve efficiency. This system routes queries based on retrieval distance and e…
-
SoftMatcha 2 enables trillion-token search in under 0.3 seconds
Researchers have developed SoftMatcha 2, a novel algorithm designed for rapid and semantically flexible pattern matching across massive text datasets. This system can search through trillions of tokens in under a second…
-
Child-directed speech aids AI language production, not comprehension
A new research paper explores how child-directed speech (CDS) impacts language models, specifically focusing on production capabilities rather than just comprehension. The study found that models trained on CDS demonstr…
-
Kronecker Embeddings slash language model parameters, boost performance
Researchers have developed Kronecker Embeddings, a novel method for representing tokens in language models that significantly reduces the number of trainable parameters. This approach replaces large embedding tables wit…
-
New Interdomain Attention Merges Transformers and SSMs
Researchers have introduced Interdomain Attention, a novel mechanism that merges the strengths of Transformers and deep state space models (SSMs). This new approach integrates an SSM into an attention module using kerne…
-
Muown optimizer improves LLM training by controlling row-norm drift
Researchers have developed Muown, a novel optimization method designed to improve the training of large language models. Muown addresses issues with the Muon optimizer, specifically the upward drift of spectral norms in…
-
OrScale optimization method improves neural network training
Researchers have introduced OrScale, a novel optimization technique designed to enhance neural network training. OrScale builds upon the Muon method by incorporating layer-wise trust-ratio scaling, which measures the Fr…
-
Researchers explore growing Transformers with modular composition and layer-wise expansion
Researchers have explored a method for training Transformer models by incrementally adding new layers to a frozen base, maintaining a constant budget for trainable parameters. This approach, termed 'Growing Transformers…
-
OpenMythos project reconstructs Anthropic's secretive Claude Mythos AI model
A new open-source project called OpenMythos has been released, aiming to theoretically reconstruct the architecture of Anthropic's Claude Mythos model. This project implements a Recurrent-Depth Transformer (RDT) with a …