LLM judges show bias in entity alignment evaluations, new study finds

A new study published on arXiv investigates the reliability of large language models (LLMs) when used as judges to evaluate entity alignment (EA) across knowledge graphs. Researchers found that LLMs exhibit significant 'anchor bias,' where the visibility of the system's decision label influences their judgment, leading to unreliable evaluations. However, a label-free protocol demonstrated that LLMs can perform reliably, especially on datasets with distinct entity names and biomedical pairs. The study also introduced a new biomedical EA benchmark and an auditing framework for LLM judge reliability. AI

IMPACT Highlights potential unreliability in LLM-based evaluation methods for structured prediction tasks, necessitating careful protocol design.

RANK_REASON The cluster contains an academic paper detailing a new study and benchmark. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• arXiv
• Entity Alignment (EA)
• Hugging Face
• Medical Subject Headings
• NeuSymEA
• SNOMED CT
• Vaibhava Lakshmi Ravideshik

• paper
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →