Natural Ungrokking

AI models forget learned rules mid-training, research finds

A new research paper introduces the concept of "natural ungrokking," describing how language models can learn a rule during pretraining, only to forget it later without any change in the loss curve. The study found that…