WSqD

New WSqD learning rate schedule offers horizon-free training for large models

Researchers have introduced WSqD, a novel learning rate schedule designed for large model training that is independent of the training horizon. Unlike existing methods like cosine annealing and Warmup-stable-decay (WSD)…