In-context learning in LLMs viewed as implicit policy gradient

A new paper proposes that in-context learning (ICL) in large language models (LLMs) can be understood as an implicit policy gradient optimization method. The research demonstrates a structural correspondence between score-conditioned ICL and policy gradient algorithms like REINFORCE, particularly when specific weight matrix configurations are met. The study also draws an analogy to KL-constrained policy optimization by deriving an upper bound on distribution shifts. Experiments across various LLMs validate these theoretical findings, showing that models effectively use score information to adjust their output distributions towards higher-scoring examples. AI

IMPACT Provides a theoretical framework for understanding how LLMs learn from examples, potentially guiding future model development and fine-tuning techniques.

RANK_REASON Academic paper detailing theoretical findings about LLM behavior. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• arXiv
• In-Context Learning
• KL-constrained policy optimization
• large-language models
• REINFORCE algorithm
• transformers

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →