Building Better Activation Oracles
Researchers have developed an improved training regimen for Activation Oracles (AOs), a method used to interpret residual stream activations in machine learning models. Their enhancements focus on using on-policy rollouts, refining conversational datasets, incorporating more layers, and optimizing the injection formula. These changes lead to substantial quality-of-life improvements for AOs and introduce AObench, the first comprehensive evaluation suite for AO quality, aiming to establish a foundation for scalable, end-to-end interpretability. AI
IMPACT Introduces a new benchmark and training improvements for AI model interpretability, potentially aiding in debugging and understanding complex models.