Megatron-LM
PulseAugur coverage of Megatron-LM — every cluster mentioning Megatron-LM across labs, papers, and developer communities, ranked by signal.
8 day(s) with sentiment data
-
KOSMOS library accelerates LLM training on AMD GPUs by up to 1.83x
A new library called KOSMOS has been developed to improve the efficiency of training large language models on GPUs. KOSMOS optimizes the communication and computation phases within transformer layers by using self-sched…
-
Lila Sciences and Anyscale scale GLM-5.3-Flash to 1M token context
Lila Sciences and Anyscale have successfully trained the open-source GLM-5.3-Flash model to handle sequences of 1 million tokens. This significant increase in context length was achieved by optimizing the training stack…
-
Sonar-MoE cuts MoE training memory by 22% and speeds up process
Researchers have developed Sonar-MoE, a novel technique for training Mixture-of-Experts (MoE) models that significantly reduces memory usage. By updating expert gradients during the backward pass and immediately freeing…
-
Grace Blackwell scheduler boosts usable GPUs by 27% through rack-locality optimization
A new scheduler, dubbed Grace Blackwell, has been developed to optimize GPU utilization for large-scale AI training and inference. This scheduler focuses on maintaining rack-locality for multi-node jobs, which is crucia…
-
Coding agents struggle with complex AI infrastructure tasks, achieving 38.9% success
InfraDev, a project evaluating coding agents, tested their ability to perform complex AI infrastructure tasks. The agents were challenged with 36 multi-file, multi-GPU changes derived from projects like Megatron-LM, vLL…
-
Mirendil launches Fern 0.1 with 10M-token context window
Mirendil has released Fern 0.1, its inaugural AI model, featuring an extensive 10 million token context window. This model is designed for AI research and development tasks, particularly for processing large codebases a…
-
HyScale system accelerates long-context LLM training with sparse attention
Researchers have developed HyScale, a new training system designed to optimize the process of training large language models with long contexts, specifically focusing on learned sparse attention mechanisms. This system …
-
Modal enables cheaper RL training with mixed-precision, mixed-accelerator samplers
Modal has developed a novel approach to reinforcement learning (RL) for large language models that utilizes mixed-precision and mixed-accelerator samplers. This method addresses the significant cost of sampling in RL by…
-
New methods tackle MoE training bottlenecks, improving efficiency and scalability
Two new research papers propose methods to optimize Mixture of Experts (MoE) models during pretraining, addressing the significant communication overhead that hinders scalability. The first paper, 'Expert Coupling in Mo…
-
New Spectral-MoE method optimizes LLM expert utilization across layers
Researchers have developed Spectral-MoE, a novel method for converting dense large language models into Mixture-of-Experts (MoE) architectures. Unlike previous approaches that use a uniform expert utilization across all…
-
MoEless framework boosts LLM serving efficiency by reducing latency and cost
Researchers have developed MoEless, a novel framework designed to improve the efficiency of serving Mixture of Experts (MoE) Large Language Models (LLMs). MoE architectures often suffer from load imbalance among experts…
-
HAPMoE planner optimizes MoE model training on heterogeneous clusters
Researchers have developed HAPMoE, a new automatic parallelism planner designed to optimize the training of Mixture-of-Experts (MoE) models on heterogeneous compute clusters. Existing methods struggle when faced with bo…
-
NVIDIA research suggests AdamW optimizer has scale ceiling
NVIDIA researchers have published new findings suggesting that the AdamW optimizer may have a scale ceiling. Their work indicates that at batch sizes up to 100 million tokens for next-token prediction, optimizers like S…
-
NVIDIA unveils Audex, a unified audio-text LLM that preserves text intelligence
NVIDIA has introduced Audex (Nemotron-Labs-Audex-30B-A3B), a unified audio-text large language model capable of understanding and generating both audio and speech. Unlike many multimodal models that experience a decline…
-
New benchmarks and methods advance multimodal LLM capabilities
Researchers are developing new methods for multimodal large language models (MLLMs) to improve their understanding of sequential audio-video data and large-scale visual recognition. One approach, DLLM-VSR, uses diffusio…
-
PyTorch tutorial simplifies distributed AI model inference
This article explains distributed inference techniques for large AI models using PyTorch. It details how to implement Data Parallelism (DP), Tensor Parallelism (TP), and Pipeline Parallelism (PP) with minimal code. The …
-
New 1.58-bit LLM family achieves 6x inference memory reduction
A new family of large language models, BitCPM-CANN, has been developed using a novel 1.58-bit ternary quantization technique. These models, ranging from 0.5B to 8B parameters, achieve significant memory reduction for in…
-
New methods tackle LLM quantization for improved efficiency and accuracy
Researchers have developed several new methods to improve the efficiency of large language models (LLMs) through quantization. OSAQ focuses on suppressing weight outliers using a low-rank Hessian property for accurate l…
-
Why Nvidia builds open models with Bryan Catanzaro
Nvidia is significantly expanding its open model program, releasing higher quality models and datasets. This strategy benefits Nvidia by capturing value from open language models, creating a sustainable advantage. The c…