logit soft-capping

New method stabilizes LLM pretraining by addressing output embedding geometry

Researchers have introduced Output Embedding Centering (OEC) as a novel strategy to stabilize the pretraining of large language models (LLMs). This method addresses the issue of output logit divergence, a common trainin…