Researchers have developed a Procrustes-conditioned Joint End-to-end Top-K Sparse Autoencoder (SAE) to extract universal features from independently trained BERT models. This method addresses the challenge of misaligned feature spaces in mechanistic interpretability by using an orthogonal Procrustes rotation before joint SAE training. The approach combines Top-K sparsity, end-to-end downstream optimization, and a dead-feature revival loss. Evaluations on five BERT model pairs across three benchmark datasets demonstrated that this pipeline yields more universal features compared to post-hoc alignment baselines, with high-universality features encoding interpretable sociolinguistic patterns. AI
IMPACT This research could improve the understanding and transferability of features learned by language models.
RANK_REASON The cluster contains an academic paper detailing a new method for model interpretability.
- arXiv
- BERT
- Hugging Face
- Procrustes-conditioned Joint End-to-end Top-K Sparse Autoencoder
- SST-2 Benchmark
- Stanford Politeness
- TweetEval Emotion
AI-generated summary · Google Gemini · from 2 sources. How we write summaries →
