Researchers have developed IFC-MCQ, an automated method for generating multiple-choice benchmarks to evaluate large language model (LLM) agents that retrieve data from Industry Foundation Classes (IFC) files. This approach addresses the challenge of costly human or LLM-based review for free-text answers by creating benchmarks with exact option-set grading. The system categorizes data into 15 categories and 36 subtypes, deriving answer keys from IFC files and constructing distractors using predefined rules or file-derived values. Evaluations using IFC-MCQ have revealed category-specific failure modes and performance variations based on the IFC model and retrieval interface, ultimately enabling scalable benchmarking for LLM-based IFC retrieval in AEC workflows. AI
IMPACT Enables more efficient and scalable evaluation of LLM agents for data retrieval in the AEC industry.
RANK_REASON The cluster describes a new method and benchmark for evaluating LLM agents on a specific data retrieval task, presented in an arXiv paper. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
