English(EN) Inter-Rater Reliability of LLM and Rule-Based Annotation for Inferential Narrative Features: Three Studies on a Turkish Corpus

LLM在土耳其语语料库研究中难以处理推断性叙事特征

作者 PulseAugur 编辑部 · [1 个来源] · 2026-09-15 04:00

一项新研究评估了大型语言模型（LLM）和基于规则的系统在土耳其语语料库中标注推断性叙事特征的评分者间信度。研究发现，包括Gemini 2.5 Flash、Grok、Claude Fable-5和ChatGPT 5.5在内的模型，以及一个基于规则的检测器，在诸如具体化隐喻等特征上与人类标注者的一致性较低。研究表明，这些推断性特征可能过于复杂，无法被当前的自动检测系统识别，或者这些定义本身尚未足够操作化，以至于任何评分者都无法一致应用。 AI

影响强调了当前LLM在细微文本分析方面的局限性，表明在理解复杂推断性特征方面存在差距。

排序理由评估LLM在特定任务上表现的学术论文。[lever_c_demoted from research: ic=1 ai=1.0]

在 arXiv cs.CL 阅读 →

AI 生成摘要 · Google Gemini · 来自 1 个来源。我们如何撰写摘要 →

本文如何被排名

Signal score: 18 / 100; Composite score across the factors below. Higher = stronger signal that this story matters right now.
Newsworthiness bucket: Tool; 评估LLM在特定任务上表现的学术论文。[lever_c_demoted from research: ic=1 ai=1.0]
Source corroboration: Single-source cluster; Only one publisher covered this so far. Single-source stories can still rank when the publisher is high-authority, but they lack cross-source corroboration.
Topics: paper, other; Editorial topic classification. Feeds into how the story surfaces on /topic/<slug> hub pages and into the per-entity coverage mix.
AI-industry relevance: High; Clearly on-topic for AI-industry coverage.
Story freshness: Breaking (< 6h); Fresh story with cross-source coverage still developing. Ranking may shift as more sources report.

完整方法见我们的编辑标准。

报道来源 [1]

arXiv cs.CL TIER_1 English(EN) · Levent Bulut · 2026-09-15 04:00

LLM与基于规则的推理叙事特征标注的评分者间一致性：基于土耳其语料库的三项研究

arXiv:2609.13936v1 Announce Type: new Abstract: Datasets that ship automatically generated feature annotations invite a question rarely asked of them: would a human agree with those labels? This report answers that for the Objective Projection corpus, a Turkish narrative dataset …

本文如何被排名

报道来源 [1]

LLM与基于规则的推理叙事特征标注的评分者间一致性：基于土耳其语料库的三项研究

相关实体

相关话题