Researchers analyze attention heads to understand in-context learning in LLMs

Researchers have developed a new framework called Task Subspace Logit Attribution (TSLA) to analyze how large language models perform in-context learning. This framework identifies specific attention heads responsible for recognizing tasks and learning from them, demonstrating their distinct roles. The study shows that these identified heads can align model states with task subspaces for recognition and rotate them for prediction, offering a unified explanation for various in-context learning mechanisms. AI

Summary written by gemini-2.5-flash-lite from 1 source. How we write summaries →

IMPACT Provides a unified, interpretable account of how LLMs perform in-context learning, potentially improving model understanding and control.

RANK_REASON Academic paper analyzing in-context learning mechanisms in large language models.

Read on arXiv cs.CL →

• arXiv
• Haolin Yang
• Task Subspace Logit Attribution
• In-Context Learning

• paper
• other