Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
Researchers have introduced Ishigaki-IDS-Bench, a new benchmark designed to evaluate the capability of large language models (LLMs) in generating Information Delivery Specification (IDS) XML from Building Information Modeling (BIM) requirements. The benchmark includes 166 expert-verified examples across various construction domains and languages, along with gold IDS files for comparison. Initial evaluations show that while LLMs can partially express information requirements, they struggle to consistently generate XML that adheres to IDS standards and IFC vocabulary constraints, with the best model achieving only 65.6% content agreement. AI
IMPACT This benchmark will help advance LLM capabilities in generating domain-specific, standardized structured data, crucial for industries like construction.