A new research paper explores the reasoning versus memorization capabilities of large language models (LLMs) when tasked with understanding and generating Chinese Xiehouyu riddles. Researchers created novel riddles to avoid data contamination and found that while frontier Chinese models showed a higher accuracy on existing riddles compared to English-centric models, suggesting larger Chinese datasets, their performance on novel riddles was still significantly lower than human experts. Gemini 3.1 Pro, however, showed a notable accuracy of 92.6% on novel riddles, surpassing human accuracy by 24%. Despite this, LLM-generated riddles received poorer ratings than human-created ones, indicating limitations in their creative abilities. AI
IMPACT Highlights potential overestimation of LLM reasoning and creativity due to data contamination, urging careful re-examination of model capabilities.
RANK_REASON Research paper published on arXiv evaluating LLM capabilities on a specific language task. [lever_c_demoted from research: ic=1 ai=1.0]
- alphaXiv
- arXiv
- CatalyzeX
- Chinese Xiehouyu Riddles
- DagsHub
- Gemini 3.1 Pro
- Gotit.pub
- Hugging Face
- human
- LLMs
- ScienceCast
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
