On the Limits of LLM-as-Judge for Scientific Novelty Assessment
A new study published on arXiv evaluates the reliability of large language models (LLMs) in assessing the novelty of scientific research questions. Researchers developed a benchmark called RQ-Bench using recent arXiv papers to compare LLM-generated questions against author-anchored reference questions. The findings indicate that LLMs consistently overestimate the novelty of generated research questions, creating a "novelty mirage" that contradicts human expert evaluations. LLMs also tend to miss crucial dimensions like narrowness or source-binding in generated questions, raising concerns about their use in scientific evaluation. AI
IMPACT Raises concerns about the current capabilities of LLMs for nuanced scientific evaluation, potentially slowing adoption in research assessment.