A new research paper introduces TKFQA, a benchmark designed to evaluate the factuality consistency and order-robustness of large language models (LLMs) when reasoning over heterogeneous knowledge sources. The benchmark comprises over 10,000 question-answering pairs grounded in tables, texts, and knowledge graphs, specifically constructed to test multi-hop reasoning chains and sensitivity to input order. Evaluations of 14 LLMs revealed significant limitations in current models' reasoning accuracy and robustness to input variations. To address this, the paper proposes ORLF, a training framework that enhances LLMs' ability to preserve knowledge-specific biases and encode topological semantics, leading to improved accuracy and reduced order-induced performance fluctuations. AI
IMPACT Highlights limitations in current LLM reasoning over complex, varied data, driving research into more robust knowledge integration.
RANK_REASON Research paper introducing a new benchmark and training framework for LLMs. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
