Researchers have evaluated the effectiveness of SHAP as an attribution method for sparse autoencoder (SAE) features in language models. Their study, using sentiment classification on the SST-2 benchmark with GPT-2 Small and a layer-7 SAE, found that SHAP and other attribution methods showed poor agreement with actual causal effects. While local per-example attribution demonstrated a directional advantage over global methods, its perceived importance was attributed to feature activity rather than causal ranking. The methods were better at detecting the presence of a feature's influence than its absence, suggesting that importance faithfulness and directional faithfulness are distinct properties requiring separate validation. AI
IMPACT Suggests current attribution methods may not reliably reflect causal influence in language model representations, necessitating new validation techniques.
RANK_REASON Academic paper evaluating an existing method (SHAP) on a specific AI task (interpreting SAE features). [lever_c_demoted from research: ic=1 ai=1.0]
- Auroc
- GPT-2 Small
- Gradient*Input
- Integrated Gradients
- multilayer perceptron
- SAE Features
- Shap
- SST-2 Benchmark
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
