mixture of experts
PulseAugur coverage of mixture of experts — every cluster mentioning mixture of experts across labs, papers, and developer communities, ranked by signal.
- instance of Mixture of Experts (MoE) 95%
- instance of arXiv 90%
- instance of alphaXiv 90%
- instance of CatalyzeX 90%
- instance of ScienceCast 90%
- instance of DagsHub 90%
- instance of large-language models 90%
- instance of Apache Software License 2.0 90%
- instance of Innu-aimun 90%
- instance of GPT OSS 20B 90%
- instance of VIDRAFT 90%
- instance of Megatron-LM 90%
- 2026-05-11 research_milestone A new paper proposes an enhanced Mixture-of-Experts framework for faster time series forecasting model training. source
25 day(s) with sentiment data
-
NVIDIA Nemotron 3.5 Super VL model now available on Amazon SageMaker JumpStart
NVIDIA's Nemotron 3.5 Super VL, a multimodal vision-language model, is now accessible through Amazon SageMaker JumpStart. This model can process images, video, and text to generate text outputs, supporting context lengt…
-
New HOPE method prunes MoE models by considering expert interactions
Researchers have developed HOPE (Higher-Order Pruning of Experts), a novel method for reducing the memory footprint of Mixture-of-Experts (MoE) language models. Unlike previous first-order methods that assess experts in…
-
Hugging Face releases comprehensive guide to Mixture of Experts models
Hugging Face has released "The Ultimate MoE Training Handbook," a comprehensive guide to training Mixture of Experts (MoE) models at scale. The handbook covers various aspects of MoE architecture, including model design…
-
InfraRSI system automates LLM training infrastructure optimization
Researchers have developed InfraRSI, a novel system designed to optimize the training infrastructure for large language models (LLMs). This agentic harness automates the process of identifying and applying training opti…
-
New $S^3$-MoE framework enhances federated learning for LLMs
Researchers have developed a new framework called $S^3$-MoE for federated learning of large language models on heterogeneous edge devices. This server-side sparse assignment approach aims to improve expert relevance est…
-
Sparse Autoencoders Reveal Fine-Grained Specialization in MoE Experts
Researchers have developed a new method using Sparse Autoencoders (SAEs) to analyze the internal workings of Mixture-of-Experts (MoE) models, specifically Qwen3-30B-A3B. This approach reveals that individual experts wit…
-
English calibration fails Lithuanian MoE models, study finds
A new study on expert-pruned Mixture-of-Experts (MoE) language models reveals that English calibration does not effectively transfer to Lithuanian. When pruning MoE models using English calibration data, Lithuanian comp…
-
New SafeStart method restores safety in faster MoE LLM serving
Researchers have developed SafeStart, a new method to mitigate safety risks associated with optimizing Mixture-of-Experts (MoE) language models for faster serving. While reducing the number of experts used per token can…
-
EDGAR framework refines Gaussian avatars using diffusion and MoE
Researchers have developed EDGAR, a new diffusion-driven framework for refining Gaussian avatars, which are used for realistic head reconstruction from images or videos. This method addresses common artifacts like blurr…
-
WeLM-Image model integrates multimodal understanding for advanced image generation
Researchers have introduced WeLM-Image, a unified image generation model designed for integration into large-scale consumer applications like WeChat. This model combines multimodal understanding with text-to-image gener…
-
New Systolic Linear Attention layer boosts TPU performance for large models
Researchers have developed Systolic Linear Attention (SLA), a new attention layer optimized for Google TPUs. Unlike existing gated-delta attention layers tuned for GPUs, SLA is designed to leverage the wide systolic mat…
-
LLM Routers, Cascades, and Mixtures Fail Differently
LLM routing architectures like routers, cascades, and mixtures have distinct failure modes that are often overlooked when focusing solely on average cost and quality. Routers can silently produce incorrect answers by mi…
-
Sonar-MoE cuts MoE training memory by 22% and speeds up process
Researchers have developed Sonar-MoE, a novel technique for training Mixture-of-Experts (MoE) models that significantly reduces memory usage. By updating expert gradients during the backward pass and immediately freeing…
-
New MoEResQ technique improves FP4 RL for Mixture-of-Experts LLMs
Researchers have identified a key challenge in applying FP4 reinforcement learning (RL) to Mixture-of-Experts (MoE) large language models. Unlike dense models, MoE models rely on a discrete routing mechanism that makes …
-
Mistral Large 4 open weights released; self-hosting requires substantial hardware
Mistral AI has released the open weights for its Mistral Large 4 model, a 1.05 trillion parameter mixture-of-experts architecture. This model, nicknamed "Le Chonk," features a 1 million token context window and supports…
-
Mixture-of-Experts models explained with Gemma, Qwen, and DeepSeek examples
This article delves into the technical underpinnings of Mixture-of-Experts (MoE) models, explaining how they achieve efficient operation. It specifically examines models like Google Gemma 4 26B, Qwen3.5–35B, Qwen3–30B, …
-
DeepSeek-V3's MoE architecture offers bandwidth savings but demands higher capacity
A recent analysis of the DeepSeek-V3 model reveals that while it has 671 billion parameters, its mixture-of-experts (MoE) architecture means it only actively uses about 36 billion parameters per token. This significant …
-
New pruning methods slash MoE LLM memory footprint · 2 sources tracked
Researchers have developed new methods for pruning Mixture-of-Experts (MoE) models to reduce their significant memory footprint. One approach, Modality-Aware Expert Pruning (MAEP), specifically addresses multimodal larg…
-
New 'structural memorization' vulnerability found in MoE LLMs
Researchers have identified a new vulnerability in Mixture-of-Experts (MoE) large language models called "structural memorization." This vulnerability allows models to memorize not only their training data but also the …
-
New research explores efficient test-time reasoning for LLMs · 10 sources tracked
Multiple research papers explore novel methods for enhancing the reasoning capabilities of large language models (LLMs) at test time. Techniques include learning when to apply teacher supervision to prevent "Teacher Dri…