Reasoning in LLMs shows little advantage in creative writing, study finds

A new arXiv paper investigates whether reasoning capabilities in large language models enhance psychological depth in creative writing. The study compared DeepSeek-R1 against DeepSeek-V3 and GPT-5 against GPT-4o using human evaluations and an LLM-as-a-Judge system. Human annotators found minimal advantage for reasoning-enhanced models, with GPT-5 slightly outperforming GPT-4o, while DeepSeek-R1 underperformed DeepSeek-V3. In contrast, the LLM-as-a-Judge exhibited a strong bias favoring reasoning, potentially obscuring the nuanced subjectivity present in human evaluations of creative tasks. AI

IMPACT Suggests that current reasoning enhancements in LLMs may not translate to improved creative output, highlighting the limitations of LLM-based evaluation for nuanced tasks.

RANK_REASON The cluster contains an academic paper published on arXiv detailing research findings on LLM capabilities. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• arXiv
• DeepSeek-R1
• DeepSeek-V3
• GPT-4o
• GPT-5
• Hugging Face
• LLM-as-a-Judge

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →