New benchmark evaluates LLMs on scientific configuration for VASP calculations

Researchers have introduced INCARBench, a new benchmark designed to evaluate the capabilities of large language models (LLMs) in configuring scientific calculations for the Vienna Ab initio Simulation Package (VASP). The benchmark includes tasks for both generating and repairing VASP input configurations. Testing 19 different LLM configurations revealed significant variations in their performance, with many models demonstrating high accuracy in semantic and policy aspects but struggling with task-critical correctness, particularly in complex scenarios involving DFT+U, magnetism, and correlated materials. AI

IMPACT Establishes a measurable capability for LLMs in scientific configuration, paving the way for more reliable AI in computational materials science.

RANK_REASON The cluster contains an academic paper introducing a new benchmark for evaluating LLMs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• arXiv
• Bin Shao
• Hugging Face
• INCARBench
• Large Language Models
• VASP

• paper
• product

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →