Researchers use program synthesis to explain transformer attention heads

Researchers have developed a novel method to approximate the behavior of attention heads in transformer language models using executable Python programs. This approach involves generating programs that can reproduce attention patterns based on input sentences and then re-ranking these programs by their predictive accuracy on held-out data. The generated programs, numbering fewer than 1,000, successfully replicated attention patterns in models like GPT-2, TinyLlama-1.1B, and Llama-3B, achieving over 75% similarity on the TinyStories dataset. Replacing a quarter of the attention heads with these programmatic surrogates resulted in only a minor perplexity increase while preserving downstream question-answering performance, offering a path toward symbolic transparency in neural models. AI

IMPACT This research offers a method for increasing the interpretability of transformer models, potentially aiding in debugging and understanding their decision-making processes.

RANK_REASON The item is an academic paper detailing a new method for interpreting neural network components. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• alphaXiv
• arXiv
• CatalyzeX
• DagsHub
• Gotit.pub
• GPT-2
• Hugging Face
• Llama 3B
• Python
• ScienceCast
• TinyLlama-1.1B
• TinyStories

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →