Kernel Machines Show Feature Learning Unnecessary for In-Context Learning

A new paper on arXiv explores in-context learning (ICL) by examining how rotation-invariant kernel machines learn ICL linear regression tasks. The study reveals that as sample size increases, kernel machines progressively learn higher-degree components of the memorizing solution, which are tied to higher-order statistics of the training data. The research suggests that generalization in ICL might be a form of underfitting the memorizing solution and that feature learning is not essential for ICL, as kernel machines can achieve generalization. AI

IMPACT Challenges the necessity of feature learning in ICL, potentially influencing future model architectures and training methodologies.

RANK_REASON Academic paper on a theoretical aspect of in-context learning. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• arXiv
• Few-shot learning
• Hugging Face
• kernel eigenbasis
• kernel machine
• linear regression
• Raventos et al.
• transformers

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →