Training
Fireworks AI has identified critical numerical parity bugs that can arise when training and serving large language models, particularly Mixture-of-Experts (MoE) architectures. These discrepancies, stemming from the non-associative nature of floating-point arithmetic and differing summation orders in distributed training versus inference, can lead to subtle but significant issues. Such drift can compromise the integrity of reinforcement learning from human feedback (RLHF) due to altered log probabilities and erode customer trust in fine-tuned models. AI
IMPACT Highlights potential issues in LLM training and serving pipelines that could affect model performance and reliability, especially for MoE architectures.