New LLM-based system automates creativity evaluation

Researchers have developed CreaEval, a novel automated creativity evaluation system designed for complex, multi-step tasks. This system decouples the traditional LLM-as-a-Judge approach into two distinct phases: memory-augmented analysis and evidence-based judging. The analysis phase uses a state-of-the-art LLM to convert multi-step responses into structured evaluation evidence, incorporating cross-step memory. The judging phase then uses this extracted evidence to provide judgments without direct access to the raw responses. Experiments indicate that CreaEval significantly improves evaluation performance across various creativity tasks. AI

IMPACT This new evaluation framework could improve the reliability and objectivity of assessing AI-generated creative content.

RANK_REASON The cluster contains a research paper detailing a new methodology for automated evaluation using LLMs. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• alphaXiv
• arXiv
• CatalyzeX
• CreaEval
• DagsHub
• Gotit.pub
• Hugging Face
• Judge-LLM
• LLM-as-a-Judge
• ScienceCast
• SoT-LLM

• paper
• other

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →