Krippendorff's alpha
PulseAugur coverage of Krippendorff's alpha — every cluster mentioning Krippendorff's alpha across labs, papers, and developer communities, ranked by signal.
4 day(s) with sentiment data
-
New method extracts scholarly claims using entity anchoring
Researchers have developed a new method for extracting structured claims from full-text scholarly articles by anchoring them to specific scientific entities like tasks, datasets, and models. This approach aims to overco…
-
New framework MLA enhances inter-annotator reliability for multi-label tasks
Researchers have introduced Multi-Label Agreement (MLA), a new framework designed to improve the evaluation of inter-annotator reliability in multi-label annotation tasks. Unlike existing metrics that are primarily for …
-
New method distills expert judgment into AI evaluation systems
A new methodology has been developed to distill expert human judgment into scalable automated evaluation systems, particularly for high-stakes domains like healthcare, finance, and politics. This approach involves assem…
-
LLM judges show increased consensus but not human alignment in new study
A new study published on arXiv explores a method to improve agreement among LLM judges when evaluating text. The research introduces a label-free procedure that identifies and rewrites rubric clauses with high disagreem…
-
New method efficiently personalizes generative user interfaces with sparse feedback
Researchers have developed a novel method for personalizing generative user interfaces (GenUIs) by learning from sparse user feedback. The approach addresses the challenge of adapting interfaces before all potential scr…
-
LLMs in mental health: Reliability, evaluation, and safety research · 4 sources tracked
A collection of research papers explores the capabilities and limitations of Large Language Models (LLMs) in mental health applications. One study evaluated Google Gemini 2.0 Flash and OpenAI ChatGPT-4o for medical diag…
-
Symbolic Augmentation boosts neural fact-checker robustness on scientific text
Researchers have developed a new method called Symbolic Augmentation to improve the accuracy of neural fact-checkers, particularly in handling scientific text. These models often struggle with numbers and units, leading…
-
New TikStance dataset enables multimodal political stance analysis on TikTok · 2 sources tracked
Researchers have introduced TikStance, a new dataset designed for analyzing political conversations on TikTok. This multimodal dataset includes 161 videos and over 13,000 comments, focusing on political figures like Don…
-
New framework uses VLMs to improve EEG-to-image reconstruction evaluation
Researchers have developed a new framework to evaluate the coherence between EEG signals and reconstructed images, addressing limitations in existing metrics like SSIM and LPIPS. This framework utilizes four Vision-Lang…
-
New method uses LLM panel to measure political positions in data-sparse regions
A new research paper proposes a novel method for measuring political positions in regions with limited data by treating large language models as individual raters within a panel. This approach aims to overcome the limit…