Cohen's kappa
PulseAugur coverage of Cohen's kappa — every cluster mentioning Cohen's kappa across labs, papers, and developer communities, ranked by signal.
10 day(s) with sentiment data
-
ImageCLEFmedical 2026 task explores GANs for synthetic medical images and privacy
The 2026 ImageCLEFmedical GANs Task, now in its fourth iteration, focuses on the privacy and security implications of generating synthetic medical images. This year's task includes three subtasks: detecting real images …
-
New benchmark for Arabic anxiety discourse released with Fanar-C-2-27B evaluation
Researchers have developed a new benchmark for analyzing Arabic anxiety-related discourse, comprising 1,000 annotated Twitter/X posts. The benchmark was created through a rigorous annotation process involving native Ara…
-
New AI benchmark tests policy reasoning, Qwen3 14B fine-tune achieves human-level performance
Researchers have developed a new benchmark called Dual-Axis Policy Entailment (DAPE) to evaluate how well AI models can reason about changes in government policies. The benchmark, which uses expert-annotated data from C…
-
AI research paper checker flags misattributed numbers
A new paper from arXiv explores the issue of misattributed numerical claims in research generated by AI agents. The study found that many AI-written papers fail integrity checks, not due to overt cheating, but because n…
-
New RISE framework improves Rhetorical Role Labeling accuracy on hard examples
Researchers have developed RISE, a novel framework designed to improve the accuracy of Rhetorical Role Labeling (RRL) for challenging text examples. This inference-time semantic reranking approach leverages the semantic…
-
New framework MLA enhances inter-annotator reliability for multi-label tasks
Researchers have introduced Multi-Label Agreement (MLA), a new framework designed to improve the evaluation of inter-annotator reliability in multi-label annotation tasks. Unlike existing metrics that are primarily for …
-
New framework Multi-Label Agreement enhances inter-annotator reliability analysis
Researchers have introduced Multi-Label Agreement (MLA), a new framework designed to evaluate inter-annotator reliability in multi-label annotation tasks. Unlike existing metrics that are primarily for single-label sett…
-
New framework analyzes creator communication on TikTok using LLMs
Researchers have developed a novel five-stage multimodal framework designed to analyze creator communication strategies on short-form video platforms like TikTok. This framework integrates various data sources, includin…
-
AI tool KAPPA automates proof-checking in statistics papers
Researchers have developed KAPPA, an open-source software system designed to automatically check proofs in statistical theory papers. KAPPA utilizes language model agents, specifically those powered by Claude Code, to t…
-
New method accurately measures AI agent skill internalization
Researchers have developed a new method to accurately measure when AI agents have internalized skills, allowing for more efficient retirement of redundant skills from skill hubs. Traditional methods that rely solely on …
-
LLM labelers misinterpret "you" as toxic, skewing conversation analysis
A new research paper highlights a significant bias in how large language models (LLMs) like Claude Haiku label online conversations as toxic. The study found that these models disproportionately flag comments containing…
-
Audit prompt evaluation graders with frozen case slices for reliable scoring
A prompt evaluation harness's grader model needs to be audited to ensure its scores are trustworthy. Teams should maintain a frozen set of adjudicated cases, separate from the grader's training data, to regularly measur…
-
New LLM-generated dataset boosts Arabic fact-checking capabilities
Researchers have introduced Arafa, a new large-scale dataset designed for fact-checking in Modern Standard Arabic. This dataset was created using a three-step automated framework that leverages large language models (LL…
-
Humans and LLMs struggle with social media sentiment analysis, study finds
A new study published on arXiv evaluates the performance of sentiment analysis tools and large language models (LLMs) on social media texts. The research found that even human annotators exhibit only fair agreement when…
-
LLM-derived weights enhance psychometric model for educational assessment reliability
Researchers have developed a new Rater Ising-Potts model that leverages Large Language Model (LLM) embeddings to assess the reliability of educational assessments. This model focuses on pairwise agreement between rating…
-
New protocol evaluates AI tutors' response to disengaged students
Researchers have developed a new protocol called Disengagement-Aware Student Simulators (DAS2) to evaluate AI tutors by modeling five learner-engagement states: engaged, gaming, wheel-spinning, off-task, and mixed. This…
-
Human evaluation remains critical for LLM quality assessment
Human evaluation is crucial for assessing Large Language Models (LLMs) because automated metrics like BLEU scores often fail to capture nuanced qualities such as coherence, creativity, and factual accuracy. This approac…
-
New Supervised Hebbian Learning Algorithm for Spiking Neural Networks Outperforms STDP
Researchers have developed a new gradient-free supervised learning algorithm for spiking neural networks (SNNs) called Supervised Spike Agreement-Dependent Plasticity (Supervised SADP). This method directly embeds class…
-
Multi-agent AI failures stem from process, not models, study finds
Multi-agent AI systems frequently fail not due to model limitations, but due to poorly defined processes and inter-agent communication issues. A study analyzing over 1,600 execution traces found that approximately 80% o…
-
Study: LLMs show limited agreement with human persuasion judgments
A new study published on arXiv investigates whether large language models (LLMs) update their beliefs in response to persuasive arguments in a manner similar to humans. The research found that LLMs exhibit only slight a…