Waïss Azizian

Pretraining data distribution shapes LLM in-context learning, study finds

A new theoretical framework and empirical study explore how the statistical properties of pretraining data influence in-context learning (ICL) in large language models. Researchers found that heavy-tailed pretraining di…