Hybrid Mamba-Transformer MoEs Hide Their Stalls in Places Dashboards Do Not Look
New hybrid Mamba-Transformer Mixture-of-Experts (MoE) models, such as NVIDIA's Nemotron 3 Nano Omni and Jamba, are exhibiting performance stalls that are not visible in standard inference dashboards. These stalls occur during the all-to-all collective communication within the MoE routing layers, which dominate the tail latency despite making up a smaller portion of the total calls. The current metrics, like GPU utilization and end-to-end latency, aggregate these issues, masking the per-layer performance variations that are crucial for optimizing inference engines. AI
IMPACT Reveals hidden performance bottlenecks in hybrid MoE models, prompting the need for new inference engine optimizations to improve latency.