Researchers have developed a new set of toy problems designed to probe how transformer models handle in-context learning (ICL) and associative recall. These problems involve training models on interleaved state observations from linear dynamical systems, requiring them to recall specific system states based on symbolic labels. The study found that models develop the ability to recall states later in training than their ability to continue predicting sequences. Mechanistic analysis suggests that next-token prediction in this context involves at least two distinct mechanisms: one for associative recall using discrete labels, and another for AI
RANK_REASON [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →