Kairong Luo

LLM pretraining research finds learning rate decay wastes best data

Researchers have identified a critical incompatibility between curriculum-based LLM pretraining and standard learning rate decay schedules. Their findings suggest that decaying learning rates can negate the benefits of …