New benchmark tests LLMs on nonlinear narrative generation

Researchers have developed a new benchmark to evaluate large language models' ability to generate nonlinear narratives, where the order of events in the story differs from the chronological order of those events. The benchmark, comprising 83 published works, includes annotations of event order, causal dependencies, and character knowledge. When tested, 19 language models showed a tendency to tell stories more chronologically than the original works, and human readers found models' requested nonlinear stories harder to follow. AI

IMPACT This research highlights current limitations in LLMs' ability to understand and generate complex narrative structures, suggesting further development is needed for more sophisticated storytelling.

RANK_REASON The cluster contains a research paper detailing a new benchmark for evaluating LLM capabilities. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• alphaXiv
• arXiv
• CatalyzeX Code Finder for Papers
• Connected Papers
• CORE Recommender
• DagsHub
• Gotit.pub
• Hugging Face
• Influence Flower
• Litmaps
• ScienceCast
• scite Smart Citations

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →