Rethinking Shrinkage Bias in LLM FP4 Pretraining: Geometric Origin, Systemic Impact, and UFP4 Recipe
A new research paper introduces UFP4, a uniform 4-bit training recipe designed to address shrinkage bias in large language model pretraining. The study identifies that current non-uniform FP4 formats, like E2M1 used in NVIDIA Blackwell/Rubin and AMD MI350 GPUs, introduce systematic rounding errors. UFP4, by contrast, utilizes uniform grids (E1M2/INT4) to improve quantization quality and demonstrates lower loss degradation on various model sizes compared to existing E2M1-based methods. AI
IMPACT This research could lead to more efficient and stable training of large language models by improving quantization techniques.