New benchmark PHITSBench tests AI's ability to generate radiation-transport simulations

Researchers have developed PHITSBench, a new benchmark designed to evaluate AI models on tasks related to the Monte Carlo Particle and Heavy Ion Transport code System (PHITS). The benchmark includes 282 tasks focused on parameter editing, syntax repair, and generating simulations from natural language descriptions. When tested, GPT-5.4 configurations showed high success rates on editing and repair tasks but struggled with generating complete simulations from scratch without domain-specific knowledge. Providing a machine-readable knowledge catalog and employing agentic workflows significantly improved performance on simulation generation, though errors persisted in physical observable selection. AI

IMPACT This benchmark could accelerate the development of AI tools for specialized scientific domains like radiation transport, improving efficiency and accuracy in research.

RANK_REASON The cluster describes a new benchmark and evaluation of AI models on a specific scientific domain, fitting the research category. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• arXiv
• GPT-5.4
• Monte Carlo Particle and Heavy Ion Transport code System
• PHITSBench

• paper
• product

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →