New benchmark reveals LLMs favor style over substance in idea evaluation

A new benchmark called SciStyleBench has been developed to diagnose and mitigate stylistic bias in Large Language Models (LLMs) when they are used to evaluate scientific ideas. The benchmark includes a three-stage evaluation environment, quantitative metrics like Style Bias Index (SBI) and Substance Recognition Rate (SRR), and a module called SciStyleExtractor designed to separate content from style. Experiments revealed that LLM judges are often swayed by writing style rather than the scientific substance of ideas, but the SciStyleExtractor module showed promise in reducing this bias and improving substance discrimination. AI

IMPACT Highlights the need for more robust evaluation methods for LLMs, particularly in scientific contexts, to ensure they assess content quality over stylistic presentation.

RANK_REASON The item is an academic paper introducing a new benchmark and methodology for evaluating LLM performance. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• Adversarial Win Rate (AWR)
• arXiv
• LLM
• SciStyleBench
• SciStyleExtractor
• Style Bias Index (SBI)
• Substance Recognition Rate (SRR)

• paper
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →