New HARP method offers training-free interpretability for LLMs

Researchers have developed a novel interpretability method called HARP (Hypothesis-driven Agentic Retrieval and Probing) for neural networks. This training-free approach utilizes an LLM agent equipped with a vector database of activations and tools for manipulation. HARP queries this database to form and validate hypotheses, outperforming existing training-based methods like Sparse Autoencoders (SAEs) in concept discovery, detection, model steering, and secret elicitation. The method's flexibility allows for on-demand indexing of new datasets, suggesting that current training-based interpretability techniques may not yet extract insights beyond their training data. AI

IMPACT This training-free interpretability method could significantly reduce the computational cost and complexity of understanding neural network behavior.

RANK_REASON The cluster contains an academic paper detailing a new research method. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• arXiv
• Hugging Face
• principal component analysis
• singular value decomposition
• Sparse Autoencoders
• Sriram Balasubramanian

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →