Sparse probes and murky physics: a case study of interpretability challenges in a foundation model for continuum dynamics
A new research paper explores the interpretability challenges of using generative AI models in scientific domains with established theories. The study focuses on the 'Walrus' foundation model for continuum dynamics, employing sparse autoencoders to analyze its internal mechanisms. Researchers found that while the model can reproduce known dynamics, its internal representations are not always consistent with established physics, leading to discrepancies in output. AI
IMPACT Highlights challenges in aligning AI model internal states with physical principles, crucial for trustworthy scientific AI.