DCLM

Researchers track attention circuit formation in 1B-class language models

A new research paper investigates the emergence of attention circuits in language models, specifically tracking how different types of attention heads form across various model architectures and training datasets. The s…