A new research paper explores the effectiveness of confidence estimates in guiding human review for LLM-generated knowledge graphs. The study, which models non-adaptive auditing as a budgeted covering problem, identifies key statistics and bounds regret for planning. Experiments on the Re-DocRED benchmark with various extraction configurations and cross-document graphs indicate that an error-rate multiplied by fan-out queue strategy leaves more errors than a joint model. However, generic LLM verification shows improvement in several configurations, and LLM adjudication helps identify potential benchmark omissions. AI
IMPACT Provides insights into improving the accuracy and efficiency of LLM-generated knowledge graphs through better human-in-the-loop processes.
RANK_REASON Research paper published on arXiv detailing methodology and experimental results. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
