AdamW
PulseAugur coverage of AdamW — every cluster mentioning AdamW across labs, papers, and developer communities, ranked by signal.
19 day(s) with sentiment data
-
New Byzantine-Robust Muon optimizer enhances distributed learning resilience
Researchers have introduced Byzantine-Robust Muon, a novel optimization method designed for distributed and federated learning environments. This method enhances robustness against adversarial behavior and data heteroge…
-
Muon optimizer complicates balancing in Mixture-of-Experts language models
A new research paper explores the balancing challenges in Mixture-of-Experts (MoE) language models, particularly when using the Muon optimizer instead of the more common AdamW. The study found that Muon, when applied to…
-
New PyTorch optimizer ApexW shows 6.3% speedup on low-end GPUs
A student developer has created a new PyTorch optimizer named ApexW, designed to reduce CUDA kernel launch overhead for faster training, particularly on lower-end GPUs. Initial benchmarks using Stable Diffusion 1.5 LoRA…
-
Nous Research fine-tunes Qwen3.8-27B to slash propaganda by 98%
Nous Research has released Arete-27B, an open-weight model fine-tuned from Alibaba's Qwen3.8-27B. This new model significantly reduces propaganda in responses, dropping it from 22% to 0.9% on Nous's internal evaluation.…
-
GraphMuon optimizer accelerates Graph Neural Network training
Researchers have developed GraphMuon, a novel optimizer designed to improve the training efficiency of Graph Neural Networks (GNNs). Unlike standard optimizers like AdamW that use coordinate-wise updates, GraphMuon adap…
-
Cost analysis reveals fine-tuning 70B models ranges from $1-2 per million tokens
Fine-tuning a 70 billion parameter model can cost between $1 and $2 per million training tokens, with a 20 million token LoRA run estimated at around $20. The cost is primarily determined by the number of tokens trained…
-
Rank-8 LoRA efficiency measured on Qwen3-4B model
A technical article details how AI models learn, focusing on the backpropagation algorithm and the AdamW optimizer. It specifically measures the performance of Rank-8 LoRA (Low-Rank Adaptation) on the Qwen3-4B model. Th…
-
LoRA fine-tuning explained: What's trained, memory use, and dataset impact
This article delves into the specifics of LoRA fine-tuning, explaining that while it trains only a small percentage of a model's parameters, the actual process is more nuanced. It details how LoRA works by adding adapte…
-
New DRAP method enhances continual learning in physical AI
Researchers have introduced DRAP (Dynamics-Risk Adaptive Plasticity), a novel method designed to improve continual learning in physical AI systems. This approach addresses the challenge of policy errors accumulating ove…
-
New NANO technique improves equifinality in hybrid AI models
Researchers have developed a new optimization technique called Non-Interfering Asymmetric Network Optimization (NANO) to address the challenge of equifinality in hybrid models. This method allows neural networks to bett…
-
Near-zero embeddings stall gated model training; new fixes proposed
A new paper identifies a critical training instability in gated sequence models, particularly those with untied embeddings. This issue, caused by near-zero input embeddings, can lead to extreme gradient amplification, e…
-
Muon optimizer research reveals complex pretraining dynamics and fine-tuning trade-offs
Recent research explores the performance of the Muon optimizer in large language model pretraining, revealing nuanced behaviors that differ from classical gradient descent theories. One study indicates that Muon's dynam…
-
New research reveals bfloat16 rounding discards most RL policy updates
A new research paper identifies a significant issue in reinforcement learning (RL) model training where the use of bfloat16 rounding silently discards most policy updates. This mismatch between training and inference, p…
-
Quantization Error Drives Language Model Training Instability
Researchers have identified a significant cause of instability during large language model pretraining: loss spikes. These spikes, often emerging late in training and termed "time-bomb spikes," are linked to the growth …
-
MUON-RT optimizer slashes LLM self-distillation memory use, boosts accuracy
Researchers have developed MUON-RT, a novel optimization technique for large language model self-distillation that eliminates the need for separate momentum buffers. By reusing the teacher-student displacement, MUON-RT …
-
AI model security risks emerge as malicious code found in hosted models · 10 sources tracked
The security of AI models is becoming a critical concern, as researchers have discovered malicious code embedded within models hosted on platforms like Hugging Face. Loading these models is akin to executing software, p…
-
LoRA outperforms QLoRA and full fine-tuning in speed and memory usage
A recent benchmark on an L40S GPU compared three fine-tuning methods for the Qwen2.5-1.5B-Instruct model: full fine-tuning, LoRA, and QLoRA. Full fine-tuning required the most memory, consuming 36.9 GiB with a micro-bat…
-
New framework enhances 3D coronary artery reconstruction from CT scans
Researchers have developed a novel framework for reconstructing 3D bilateral coronary artery trees from cardiac CT angiography (CCTA) data. This method addresses limitations such as quantization, territory ambiguity, an…
-
New LITE strategy accelerates LLM pre-training by enhancing flat-direction dynamics
Researchers have developed a new acceleration strategy called LITE to improve the efficiency of Large Language Model (LLM) pre-training. LITE enhances training dynamics by applying larger damping coefficients and learni…
-
Developer releases ApexW PyTorch optimizer for large GPU workloads
A developer has created a new PyTorch optimizer called ApexW, designed to reduce memory usage and speed up optimizer steps for large GPU workloads. Early benchmarks show ApexW can be up to 1.5x faster than PyTorch's fus…