Frontier LLMs show widespread verbatim retrieval of molecular data, study finds

A new study published on arXiv investigates the phenomenon of "molecular déjà vu" in frontier language models, where models appear to retrieve published molecular property values verbatim rather than predicting them. The research found that this verbatim retrieval is widespread but varies significantly across different regression benchmarks. Interestingly, the rate of retrieval increased substantially when models were prompted with higher reasoning levels. The study also explored methods to interrupt this retrieval, suggesting that a model's general predictive capability is not solely determined by memorized values. AI

IMPACT This research highlights potential limitations in LLM evaluation, suggesting that current benchmarks may not fully capture true predictive capabilities due to memorization.

RANK_REASON The cluster contains an academic paper detailing research findings on language models. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• arXiv
• Smiles

• paper
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →