A new arXiv paper details how different numerical state tracing methods can lead to behavioral divergence in sparse Mixture-of-Experts (MoE) models, even when mathematically equivalent. Researchers found that varying aggregation semantics in DeepSeek-V4 Flash, while keeping MoE state frozen, resulted in distinct execution paths and output continuations. The study highlights that identical tokens do not guarantee identical autoregressive states, as divergence can emerge later in the generation process. These findings suggest that operand conversion, accumulator precision, and reduction order are critical components of a numerical compatibility contract for MoE runtimes and hardware. AI
IMPACT Highlights potential issues in MoE model reproducibility and the need for standardized numerical compatibility contracts.
RANK_REASON Academic paper detailing novel findings on model behavior. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
