Researchers have introduced GeoOutageBench, a new benchmark designed to evaluate Large Language Models (LLMs) in their ability to perform geospatiotemporal Knowledge Graph Question Answering (KGQA) for analyzing power outages and infrastructure resilience. This benchmark uniquely integrates visual, textual, and structured data from various sources, including outage records, satellite imagery, and weather data, along with domain ontologies. GeoOutageBench offers a query taxonomy with varying difficulty levels to assess LLMs' understanding of ambiguous spatiotemporal questions, their use of ontologies, and their accuracy in multimodal KGQA retrieval, providing a foundation for assessing LLM-KG systems for real-world infrastructure analysis. AI
IMPACT This benchmark could lead to more robust LLM systems for critical infrastructure analysis and disaster response.
RANK_REASON The item describes a new benchmark for evaluating LLMs in a specific research area (geospatiotemporal KGQA), presented in an arXiv paper. [lever_c_demoted from research: ic=1 ai=1.0]
Read on arXiv cs.IR (Information Retrieval) →
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
