item response theory
PulseAugur coverage of item response theory — every cluster mentioning item response theory across labs, papers, and developer communities, ranked by signal.
-
AI time horizon growth may be curve-fitting artifact, study finds · 4 sources tracked
A new statistical analysis of the METR plot, which measures AI capabilities by the time it takes humans to complete tasks that AI can solve with 50% probability, suggests that the commonly cited exponential growth in AI…
-
Reverse Item Response Theory Applied to Cancer Drug-Response Data
Researchers have developed a novel application of reverse Item Response Theory (IRT) to analyze fragmented cancer drug-response data. This method treats cancer types as latent subjects with resistance abilities and drug…
-
New method explains natural language question difficulty using LLMs
Researchers have developed a novel data-driven method to automatically generate and validate natural-language explanations for why certain questions are more difficult than others. This approach utilizes Item Response T…
-
New method drastically cuts AI safety benchmark costs
Researchers have developed a more efficient method for evaluating the safety of language models using Item Response Theory (IRT). This psychometric approach, applied to six common safety benchmarks, demonstrates that IR…
-
LLM preference generation and calibration research shows model discordance and improved methods
Recent research explores the reliability and calibration of Large Language Model (LLM) generated preferences. One study found that while LLMs exhibit self-coherence in preferences, significant discordance exists across …
-
New QC-Stark Benchmark Reveals LLM Capability Gaps in Quantum Computing
Researchers have developed QC-Stark, a new benchmark designed to evaluate large language models (LLMs) on 11 distinct quantum computing tasks. These tasks cover areas such as circuit construction, debugging, compilation…
-
Rubric Response Theory enhances RL rewards with item response modeling
Researchers have introduced Rubric Response Theory (RRT), a novel approach for generating rewards in reinforcement learning tasks where human judgment is required. Unlike traditional methods that sum points from rubric …
-
IXPLORE algorithm enhances political data analysis with improved accuracy
Researchers have developed IXPLORE, a new algorithm for ideal point estimation that combines predictive accuracy with a sparsity-aware likelihood function. This approach aims to improve the analysis and visualization of…
-
MMLU benchmark audit reveals focus on retrieval over reasoning
A new audit of the Massive Multitask Language Understanding (MMLU) benchmark reveals that its aggregate score primarily measures factual retrieval rather than reasoning capabilities. Researchers used Item Response Theor…
-
BenchMIRT method reveals what LLM benchmarks truly measure · 2 sources tracked
Researchers have introduced BenchMIRT, a novel methodology designed to dissect the performance of large language models (LLMs) on benchmarks by analyzing individual prompts. This approach, inspired by Item Response Theo…
-
New causal modeling approach enhances student competency assessment
Researchers propose a new approach to student competency assessment using structural causal modeling, moving beyond traditional psychometric models like item response theory. This method aims to explicitly support inter…
-
New approach to educational assessment uses causal modeling
This paper proposes a structural causal modeling approach for educational assessment, moving beyond traditional item response theory. The authors advocate for a framework that explicitly supports interventional and coun…
-
New IRT Framework Unveils Cross-Lingual Safety Gaps in LLMs
A new research paper proposes a novel framework to analyze why large language models' safety guardrails falter in non-English languages. The proposed Multi-Group Item Response Theory (IRT) model, named MultiJail, aims t…
-
LLM calibration research proposes new methods for benchmark comparability and out-of-domain generalization
Two new research papers propose methods to improve the calibration of large language models (LLMs). The first paper introduces a framework based on Item Response Theory (IRT) that uses anchor items to calibrate new benc…
-
LLM generalization across difficulty levels is limited, new study finds
A new research paper published on arXiv investigates the generalization capabilities of large language models (LLMs) across varying task difficulties. The study, which utilized Item Response Theory (IRT) and LLM outputs…
-
LLMs struggle to accurately gauge item difficulty, underestimating complex learner challenges
Recent research indicates that while large language models (LLMs) can predict item difficulty levels with moderate accuracy, they struggle with identifying truly hard items. Studies found that LLMs tend to underestimate…
-
New method quantifies uncertainty in LLM benchmarks
Researchers have developed Laplace-PSN-IRT, a new method to quantify uncertainty in Large Language Model (LLM) benchmarks. This approach uses a post-hoc Laplace approximation to provide Bayesian posterior inference with…
-
AI agent evolution and benchmark rankings face scrutiny · 2 sources tracked
A new arXiv paper suggests that automatically evolving AI agent scaffolding does not consistently outperform simple search methods, showing limited generalization to new tasks. Separately, research indicates that Item R…
-
New research explores adaptive LLM evaluation and self-improvement techniques · 10 sources tracked
Researchers are developing new methods to evaluate and improve large language models (LLMs). One approach, ATLAS, uses item response theory to significantly reduce the number of items needed for accurate LLM evaluation,…
-
New benchmarks evaluate Portuguese text embedding models, revealing performance gaps
Two new benchmarks, MTEB-PT and MTEB-PT (Brazilian Portuguese), have been released to evaluate text embedding models specifically for the Portuguese language. These benchmarks address the underrepresentation of Portugue…