Kaiyue Wen

Study finds AdamW dominance challenged by other optimizers, with caveats

A new study published on arXiv systematically compares ten deep learning optimizers for language model pretraining, challenging the dominance of AdamW. The research highlights that previous comparisons were often flawed…