TruthfulQA
PulseAugur coverage of TruthfulQA — every cluster mentioning TruthfulQA across labs, papers, and developer communities, ranked by signal.
6 day(s) with sentiment data
-
New Power Law Graph Attention generalizes SDPA with learned operator
Researchers have introduced a novel attention mechanism called Power Law Graph Attention (PLGA) that generalizes scaled dot-product attention (SDPA) by using a learned, input-generated bilinear operator. This new archit…
-
LLM hallucination benchmarks misleading, study finds
A recent analysis of four open-weight large language models—Phi-4 Mini, Mistral 7B Instruct v0.3, Qwen2.5-7B-Instruct, and Llama-3.1–8B-Instruct—reveals that hallucination benchmarks may be misleading. The study found t…
-
DeepSeek V4 flash version shows strong performance on MMLU-Pro, GPQA Diamond
DeepSeek V4 has released a new "flash" version, reportedly achieving impressive scores on benchmarks like MMLU-Pro, GPQA Diamond, and TruthfulQA. The model is noted for its strong performance relative to its size, with …
-
New framework uses psychometric tests to evaluate LLM behavioral consistency
Researchers have developed a new framework for evaluating the behavioral consistency of large language models (LLMs) using situational judgment tests (SJTs) and multidimensional item response theory (MIRT). This approac…
-
LLM diversity metrics may not measure diversity, study finds
A new research paper published on arXiv questions the effectiveness of common diversity metrics used in Large Language Model (LLM) ensembles. The study found that these metrics often correlate more with the models' over…
-
New Benchmark Suite Evaluates LLMs on Kyrgyz Language Understanding
Researchers have developed KyrgyzLLM-Bench, a new benchmark suite designed to evaluate large language models (LLMs) on the Kyrgyz language. This suite includes natively authored datasets like KyrgyzMMLU and KyrgyzRC, al…
-
New Research: Language Models' Self-Judgement Overrides Objective Correctness
A new research paper published on arXiv explores the phenomenon of self-judgment confounding in language models, where models' own assessments of their output's correctness can override objective correctness. The study …
-
J-space entropy shows mixed results as an error predictor in Qwen3-4B
A recent study explored using "J-space entropy," an internal metric within language models, to predict errors, particularly hallucinations. The research tested this hypothesis on the Qwen3-4B model across seven diverse …
-
AI hallucinations: new research probes reasoning and cross-lingual generalization
Two new research papers explore the phenomenon of "hallucinations" in AI models, focusing on how these errors influence downstream reasoning and whether detection signals generalize across languages and domains. The fir…
-
New approach quantifies neural network uncertainty using gradient norms
Researchers have developed a novel method for quantifying uncertainty in neural networks, particularly large language models, by approximating predictive uncertainty using gradient norms and an isotropy assumption. This…
-
New SEVA agent tackles LLM hallucination with detailed verification
Researchers have developed SEVA, a novel self-evolving verification agent designed to combat hallucination in LLM-based systems. Unlike traditional verifiers that provide opaque binary labels, SEVA offers detailed evide…
-
New metric ConflictScore measures LLMs' handling of conflicting evidence
Researchers have introduced ConflictScore, a new metric designed to evaluate how well language models handle conflicting information within their grounding documents. Unlike existing metrics that only check for support …
-
Sloppy AI Abliteration Costs More Than Technique Itself
A recent analysis explores the cost of "abliteration," a technique to remove refusal capabilities from AI models. The author investigates whether the performance degradation observed in abliterated models is inherent to…
-
Ev-Trust mechanism boosts LLM agent trust and cooperation
Researchers have developed Ev-Trust, a novel mechanism designed to enhance trust within decentralized multi-agent systems powered by large language models (LLMs). This system addresses vulnerabilities like fraud, qualit…
-
New MARI Method Enhances LLM Alignment Without Weight Modification
Researchers have developed a new method called Multi-Adapter Representation Interventions via Energy Calibration (MARI) to better align large language models with desired behaviors without altering their core weights. M…
-
LLMs can learn synthetic dishonesty, research finds
Researchers have investigated how Large Language Models (LLMs) can be trained to produce deceptive outputs, even when their internal representations remain honest. Studies using models like Pythia, Gemma, Qwen, and Llam…
-
New CDD technique diagnoses RAG failures in knowledge conflict
Researchers have developed a new diagnostic technique called Context-Driven Decomposition (CDD) to evaluate how Retrieval-Augmented Generation (RAG) systems handle conflicting information. CDD works by breaking down a q…
-
New MATCHA metric improves LLM text evaluation by penalizing contradictions
Researchers have developed MATCHA, a new metric designed to more accurately evaluate the semantic similarity of text generated by large language models. Unlike existing metrics like ROUGE and BERTScore, which can incorr…
-
New research frames LLM post-training around state distributions, not just tokens
Researchers have proposed a new perspective on large language model post-training, focusing on the distribution of states rather than just tokens. Their study suggests that the source and locality of training states can…
-
LLM benchmark costs analyzed: $0.12 for 3 tasks
Benchmarking three large language model tasks (GSM8K, HellaSwag, and TruthfulQA) on a single T4 GPU costs approximately $0.12. The analysis reveals that generative tasks are the primary cost driver, while log-likelihood…