FineWeb-Edu
PulseAugur coverage of FineWeb-Edu — every cluster mentioning FineWeb-Edu across labs, papers, and developer communities, ranked by signal.
6 day(s) with sentiment data
-
New research analyzes Z-loss backward geometry in language models
A new paper analyzes Z-loss, a technique used to stabilize language model training, from a backward-pass perspective. The research introduces a "backward-transport" view that separates the Z-loss source from the archite…
-
Minibatch persistency offers data savings, not speed or energy, study finds
A new study published on arXiv investigates the impact of minibatch persistency, a technique that reuses data across multiple optimization steps rather than drawing fresh data each time. The research found that minibatc…
-
Deeper, thinner models outperform wider ones in sub-150M parameter regime
Researchers have explored the impact of model depth versus width in the sub-150 million parameter range, finding that a deeper, thinner architecture (23 layers x 576 hidden) outperformed a wider, shallower one (53.5M vs…
-
New pipeline scores educational data for LLM pre-training
Researchers have developed Edu-QuRating, a new pipeline for multi-dimensional educational data scoring and curation. This system defines education-specific rubrics and uses an LLM judge to label document pairs, distilli…
-
LLM mid-training data composition impacts alignment, research finds
A new research paper explores the optimal data composition during the mid-training phase of large language models, finding that a moderate band of data (10%-40%) is best for all domains. The study, which used the Qwen3-…
-
New Grouped Value Attention method slashes Transformer KV cache size
Researchers have introduced Grouped Value Attention (GVA), a novel method to reduce the memory footprint of KV caches in Transformer models. GVA stores grouped values and reconstructs keys using a learned linear map, wh…
-
Graph Machine architecture offers efficient LLM pretraining alternative · 2 sources tracked
Researchers have introduced a novel architecture called Graph Machine (GM) that aims to improve pretraining efficiency for large language models. GM utilizes sparse dynamic routing and a pointer-chasing mechanism to mai…
-
OrScale optimization method enhances neural network training
Researchers have introduced OrScale, a novel optimization method designed to improve the training of large neural networks. OrScale addresses the direction and magnitude of updates by adapting the trust-ratio principle …
-
TANGO model introduces novel gating operators for language modeling
Researchers have introduced TANGO, a novel language modeling architecture that aggregates token information through nonlinear gating operators. This approach replaces standard Transformer components with a single cross-…
-
New DeltaMomentum optimizer speeds up deep learning training
Researchers have introduced DeltaMomentum, a novel approach to updating momentum in deep learning optimizers. Unlike traditional methods that use a fixed rate for exponential moving averages, DeltaMomentum dynamically a…
-
User trains 1.1B LLM from scratch for $200, shares code and model
A user has successfully trained a 1.1 billion parameter large language model from scratch for approximately $200. The model, named 'gemmeh', was pre-trained on 20 billion tokens from the fineweb-edu dataset and then fin…
-
New trust-region framework enhances adaptive moment estimation in optimization
Researchers have introduced a novel trust-region framework designed to analyze the behavior of adaptive moment estimation methods in stochastic gradient optimization. This framework constrains the magnitude of update st…
-
New CuraWeb corpus boosts LLM performance with optimized data curation
Researchers have developed CuraWeb, a new 2 trillion token English corpus designed to improve the pretraining data for large language models. Unlike previous methods that focused on singular optimization objectives, Cur…
-
Möbius RoPE enhances in-context retrieval reliability in language models
Researchers have developed a new positional encoding technique called Möbius RoPE, which utilizes anti-periodic boundary conditions to improve in-context retrieval reliability in language models. This method, applied to…
-
User trains 0.5M parameter model Silia-v2 on 1B tokens
A user has trained a small language model with 0.5 million parameters on 1 billion tokens from the FineWeb-Edu dataset. This model, named Silia-v2, is an iteration on a previous research paper and incorporates architect…
-
Researchers analyze Transformer attention mechanisms and feed-forward networks · 2 sources tracked
Two new research papers explore the fundamental components of Transformer models, specifically focusing on the role of attention mechanisms versus feed-forward networks. The first paper, "A Controlled Study of Attention…
-
Depth-Recurrent Transformers Show Per-Token Fixed-Point Convergence
Researchers have investigated the internal computations of depth-recurrent transformers, specifically how each token's state evolves over multiple processing loops. They found that the recurrent state converges to a fix…
-
Small language model trained on single GPU detailed in new study
Researchers have detailed a method for training a small language model, L20-Edu-135M, using significantly fewer computational resources, specifically on a single NVIDIA L20 GPU. The study focused on data efficiency, uti…
-
New pretraining method enhances LLM safety with integrated reflection
Researchers have introduced a new method called Safety Reflection Pretraining, designed to enhance the safety alignment of large language models (LLMs) during the pretraining phase. This approach goes beyond simply filt…
-
EverydayGPT uses confidence gating to cut RAG latency by 120x
Researchers have developed EverydayGPT, a conversational question-answering system that uses a Confidence-Gated Routing (CGR) mechanism to improve efficiency. This system routes queries based on retrieval distance and e…