Sparse autoencoders show reward filtering captures solution completeness, not reasoning quality

Researchers have developed a reward-informed sparse autoencoder (RI-SAE) to interpret language model activations, specifically focusing on reasoning capabilities. While the RI-SAE successfully separated high-reward and low-reward reasoning trajectories in Llama-3.1-8b, further analysis revealed that this separation was primarily due to solution completeness rather than genuine reasoning quality. Control experiments showed that even a simple TF-IDF classifier and structural cues like answer boxing could achieve similar discriminatory power, suggesting that reward filtering is an effective but superficial method for interpretability. AI

IMPACT This research suggests that current interpretability methods using reward signals may not fully capture reasoning quality, potentially impacting how we evaluate and understand AI behavior.

RANK_REASON This is a research paper detailing a new method for interpreting language models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• arXiv
• GRPO
• Hugging Face
• Llama-3.1:8b
• Reward-Informed Sparse Autoencoder
• Sparse Autoencoders
• tf–idf

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →