New ROTATE method disentangles MLP neuron weights in language models

Researchers have developed a new method called ROTATE (Rotation-Optimized Token Alignment in weighT spacE) to better understand the information encoded within the weights of large language models. This data-free technique analyzes neuron weights directly, identifying concepts by looking for high kurtosis in their vocabulary space projections. Experiments on Llama 3.1 8B-Instruct and Gemma 2-2B-it showed that ROTATE successfully isolates interpretable directions, termed vocabulary channels, which can be used to selectively disable specific neuron behaviors or concepts. The method offers a scalable way to decompose neuron weights, potentially improving the interpretability of language models. AI

IMPACT Provides a new tool for understanding internal model mechanisms, potentially aiding in debugging and improving LLM interpretability.

RANK_REASON The cluster contains a research paper detailing a new method for interpreting language model weights. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• alphaXiv
• arXiv
• Asaf Avrahamy
• CatalyzeX
• DagsHub
• Gemma 2-2B-it
• Gotit.pub
• Hugging Face
• Llama 3.1 8B-Instruct
• ROTATE
• ScienceCast

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →