oMeBench: Towards Robust Benchmarking of LLMs in Organic Mechanism Elucidation and Reasoning
Researchers have introduced oMeBench, a new benchmark designed to evaluate the organic mechanism reasoning capabilities of large language models. The benchmark includes over 10,000 annotated mechanistic steps and a dynamic evaluation framework called oMeS for fine-grained scoring. Initial analysis reveals that while current LLMs show some chemical intuition, they struggle with consistent multi-step reasoning, though fine-tuning on the dataset significantly improved performance. AI
IMPACT This benchmark could drive the development of LLMs with more robust scientific reasoning abilities, particularly in chemistry.