New benchmark reveals LLMs exhibit significant framing bias in news summaries

Researchers have developed a new benchmark called Frame In, Frame Out (FIFO) to measure framing bias in news summaries generated by large language models. The benchmark, which includes over 15,000 jury-annotated examples, found that LLM-generated summaries often exhibit higher framing rates than human-written ones. This bias was particularly pronounced in summaries related to science and public health, highlighting framing as a critical but often overlooked aspect of summarization quality. AI

IMPACT Highlights a new evaluation metric for LLM-generated text, potentially influencing future model development and deployment in news summarization.

RANK_REASON The cluster describes a new academic paper introducing a novel benchmark for evaluating LLM-generated content. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• Frame In, Frame Out (FIFO)
• Large language models
• Valeria Pastorino
• XSum dataset

• paper
• safety

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →