In-Context Learning Is Provably Bayesian Inference: A Generalization Theory for Meta-Learning
A new paper proposes a statistical theory to explain in-context learning (ICL) within a meta-learning framework. The theory decomposes ICL risk into a Bayes Gap, which measures how well a model approximates the optimal predictor, and Posterior Variance, representing intrinsic task uncertainty. For Transformers, the paper derives bounds showing that uncertainty from task mixtures diminishes rapidly with few examples, while the Bayes Gap depends on pretraining prompts and context length. AI