Adam
PulseAugur coverage of Adam — every cluster mentioning Adam across labs, papers, and developer communities, ranked by signal.
- used by muon 90%
- instance of alphaXiv 90%
- instance of SGD 90%
- used by stochastic gradient descent 90%
- instance of Nadam 90%
- developed AdaGrad 90%
- instance of AdaGrad 90%
- used by transformers 80%
- competes with gradient descent 80%
- competes with muon 70%
- instance of DagsHub 70%
- used by ScienceCast 70%
17 day(s) with sentiment data
-
RLVR research shows reward hacking can be amplified or suppressed
A new research paper explores how reinforcement learning with verifiable rewards (RLVR) interacts with imperfect verifiers, revealing that reward hacking can be both suppressed and amplified. The study found that RLVR a…
-
GraphMuon optimizer accelerates Graph Neural Network training
Researchers have developed GraphMuon, a novel optimizer designed to improve the training efficiency of Graph Neural Networks (GNNs). Unlike standard optimizers like AdamW that use coordinate-wise updates, GraphMuon adap…
-
Islamic texts highlight knowledge's role in society and faith
This article discusses the profound importance of knowledge and science in building societies, drawing heavily on Islamic theological and philosophical perspectives. It posits that individuals with knowledge are superio…
-
New bilevel optimization method achieves optimal convergence rates
Researchers have developed a new method for bilevel optimization, a technique crucial for hyperparameter tuning in computational imaging. The proposed approach, Strategically Inexact Stochastic Gradient Descent, address…
-
Federated learning models can learn from static context for time series forecasting
A new research paper explores how federated learning models can effectively utilize static contextual information for time series forecasting. The study found that per-batch aggregation methods like FedAdam are superior…
-
New IFT method for throttle valve control faces bias challenges
Researchers have developed a new method called Iterative Feedback Tuning (IFT) to optimize model-free control parameters, particularly for automotive electronic throttle valves. This technique uses stochastic gradient d…
-
Quantum computing principles applied to anomaly detection in new research
Two new research papers explore the application of quantum computing principles to anomaly detection. The first paper, "Quantum-Inspired Tensor Network Autoencoders for Anomaly Detection," proposes a MERA-based autoenco…
-
New FARO optimizer improves LLM training by reallocating risk budgets
Researchers have introduced FARO, a novel optimization framework designed to improve the training of large language models. FARO adapts the optimization process by treating parameter blocks as the fundamental unit, dyna…
-
Neural network optimizer names cause search headaches for ML researchers
A discussion on Reddit's r/MachineLearning subreddit highlights the naming conventions of neural network optimizers. Users are expressing frustration with the often unusual and non-unique names of these optimizers, such…
-
Neural network fits in 510 bytes, dreams Linux boot logs
A developer has successfully created a neural network that fits within a 510-byte boot sector and can hallucinate Linux boot logs. The model, trained on synthetic dmesg data, uses 11,520 bytes for its weights and operat…
-
Research paper questions effectiveness of excessive planning in AI world models
A new research paper titled "When More Planning Hurts: Learned Planners for World Models" explores the effectiveness of learned planners in world models for robotics. The study, authored by Josef Liyanjun Chen and other…
-
Researchers identify 'precision wall' impacting AI optimization algorithms
Researchers have identified a "precision wall" that affects optimization algorithms when dealing with power-law measurement spectra in low-rank matrix sensing. This wall causes first-order optimizers like Adam to stall …
-
Muon optimizer analysis reveals performance nuances vs. SignSGD
A new research paper titled "Phases of Muon: When Muon Eclipses SignSGD" analyzes the performance and behavior of Muon, a spectral optimizer, in comparison to SignSGD and Adam. The study derives explicit deterministic d…
-
Error Diffusion optimizer enables efficient training of binary-weight language models
Researchers have developed a novel method called Error Diffusion (ED) to train binary-weight language models more efficiently. This technique significantly reduces the memory required for training by replacing full-prec…
-
New FA-GC Method Enhances Federated Learning with Adaptive Optimizers
Researchers have introduced Federated Adaptive Gradient Correction (FA-GC), a new method designed to improve federated learning with adaptive optimizers like Adam. Unlike previous approaches that correct gradients only …
-
AI research papers propose new methods for evaluation and gradient optimization
Two new research papers explore methods for optimizing the evaluation process in AI development. The first paper, "Where Should the Evaluation Budget Go in LLM-Driven Agent Scaffold Evolution?", proposes a framework cal…
-
New UTA method optimizes learning rate decay for AI models
Researchers from the University of Texas at Austin have developed a new learning rate decay method called Universal Training Adaptive (UTA) LRD. This method aims to improve model training by being adaptive to the traini…
-
New analysis reveals why AI models forget during updates
Researchers have developed a new functional-path analysis to understand why supervised fine-tuning (SFT) of language models can lead to forgetting previously acquired behaviors. This analysis decomposes changes in old-l…
-
Researchers Fix Numerical Bugs in Softmax and Cross-Entropy Implementations
Researchers have identified and fixed two numerical bugs in standard implementations of softmax, log-softmax, and cross-entropy. The first bug, Softmax Collapse, causes the target logit's gradient to round to zero when …
-
New continuous-state probabilistic computer hardware demonstrated
Researchers have demonstrated a novel continuous-state probabilistic computer (CPC) that utilizes programmable energy-defined sampling for complex problem-solving. Unlike traditional binary systems, this CPC handles con…