Researchers have introduced Ekphrasis, a new benchmark designed to evaluate the visual creative ideation capabilities of text-only language models. This benchmark measures a model's ability to generate useful, expressive, and novel textual visual plans, separating these qualities from mere fluency. Ekphrasis comprises 400 tasks across abstraction, combination, transformation, and adaptation, using pairwise comparisons and Bradley-Terry models to aggregate preferences and identify population clichés. A cross-modal grounding study indicates that the ordering of visual ideation capabilities identified by Ekphrasis largely holds true even after faithful rendering and blind image-level preference judgments, suggesting its effectiveness in assessing visual ideation beyond prose quality. AI
IMPACT This benchmark could lead to more nuanced evaluations of LLMs' creative potential beyond text generation.
RANK_REASON The cluster contains an academic paper introducing a new benchmark and methodology for evaluating language models. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
