Chemical language models' internal representations analyzed with sparse autoencoders

A new research paper explores the internal workings of chemical language models (cLMs) by applying sparse autoencoders (SAEs) to MolFormer. The study reveals that early layers of the model focus on syntactic patterns and position tracking, while later layers capture more meaningful semantic information, including pharmacologically relevant features. The research also found that non-canonical SMILES strings cause greater disruption to the model's representations than invalid SMILES, highlighting the importance of input format. To facilitate further investigation, the authors developed InterMol, an interactive tool for visualizing SAE activations. AI

IMPACT Provides insights into how chemical language models process molecular data, potentially improving their design and application in chemistry.

RANK_REASON Research paper analyzing a specific model's internal representations. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• alphaXiv
• arXiv
• CatalyzeX
• Christian Kenneth Bowring Melville
• CORE Recommender
• DagsHub
• Gotit.pub
• Hugging Face
• IArxiv Recommender
• intermolecular force
• MoLFormer
• ScienceCast
• SMILES
• Sparse Autoencoders

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →