GPQA: A Graduate-Level Google-Proof Q&A Benchmark
PulseAugur coverage of GPQA: A Graduate-Level Google-Proof Q&A Benchmark — every cluster mentioning GPQA: A Graduate-Level Google-Proof Q&A Benchmark across labs, papers, and developer communities, ranked by signal.
- instance of Humanity's Last Exam 90%
- instance of long-context reasoning 90%
- instance of GLM-5.2 90%
- instance of MATH500 90%
- instance of LiveCodeBench 90%
- instance of MMLU-Pro 70%
- competes with MMLU-Pro 70%
- instance of SciCode 70%
- used by GSM8K 70%
- instance of HumanEval 70%
- instance of VIDRAFT 70%
- used by MATH500 70%
19 day(s) with sentiment data
-
AI model evaluation evolves from token prediction to complex workflows
The evaluation of frontier AI models has evolved through distinct eras, moving from simple token prediction and log-likelihood grading to assessing instruction following, then to verifying math, code, factuality, and sa…
-
New LLM routing policy maximizes task completion within fixed budgets
Researchers have developed a new contextual bandit policy called ADP-HT designed to optimize the allocation of tasks to large language models (LLMs) within fixed budgets. This policy learns task-specific success probabi…
-
Liquid AI unveils efficient 1GB LFM3 model family for local agents
Liquid AI has released LFM3, a new family of open-weight models designed for efficiency and local deployment. The flagship LFM3-1.7B model, weighing approximately 1 GB, boasts a 256K-token context window and a unique tr…
-
New ADP-HT policy optimizes LLM use within fixed budgets
Researchers have developed a new contextual bandit policy called ADP-HT to optimize the use of large language models (LLMs) within fixed budgets. This policy aims to maximize the number of tasks solved before a budget i…
-
LLM evaluation flawed by minor weight changes and truncation, study finds
A recent analysis revealed that even minor variations in model weights, as small as 0.005%, can lead to significant score differences of up to 1.4%p in LLM evaluations. The study found that repeating the same evaluation…
-
LLM benchmarks reveal performance across GPQA, SciCode, and reasoning tasks
Independent benchmarks reveal the performance of several large language models across various metrics. Muse Spark Open achieved 89% on GPQA and 58.4% on SciCode, while Muse Glimmer scored 83.5% on GPQA and 44.9% on SciC…
-
DeepSeek V4 Pro 0813 (Max) achieves high scores on GPQA and long context tasks
DeepSeek V4 Pro 0813 (Max) has achieved a 92.8% score on the GPQA benchmark and demonstrated 80.3% performance in long context tasks. The model also achieved 41% on the HLE benchmark and processed information at 76.9 to…
-
Your audit log has 1,509 "allowed" rows. That is not evidence.
Two recent articles highlight critical distinctions between audit logs and model reasoning traces, arguing that the latter are insufficient for genuine evidence or security. The first piece emphasizes that an audit log …
-
UC Berkeley's LEANN vector index slashes RAG storage needs from 201GB to 6GB
Researchers at UC Berkeley's Sky Computing Lab have developed LEANN, an open-source vector index that significantly reduces storage requirements for RAG systems. By recomputing embeddings on demand rather than storing t…
-
DeepSeek R1 0528 achieves 81.3% on GPQA benchmark
DeepSeek R1 0528, a model released in May 2025, achieved an 81.3% score on the GPQA benchmark. The model's performance also indicates a cost-effectiveness of 6.3 intelligence points per dollar spent. Further details on …
-
Unofficial Qwen3.8 derivative shows benchmark gains, retains CVSS scoring issues
An unofficial derivative model, autotrust/Qwen3.8-Flash-Next-E384-FP4-V2, has been released, built upon Alibaba Group's Qwen3.8-Flash-Next-E384-FP4-V2. This model incorporates GRPO-style reinforcement learning and is va…
-
Hy3-preview model shows significant gap on GPQA vs HLE benchmarks
An independent measurement shows the Hy3-preview model achieving 73.2% on the GPQA benchmark, significantly outperforming the 7% score on HLE. This substantial difference highlights a key area for further investigation …
-
New site compares LLM benchmarks, pricing, and VRAM needs
A new website, modelanalysis.si, has been launched to provide a centralized platform for comparing large language models. The site offers detailed comparisons of benchmarks, including real-world metrics like SWE-bench V…
-
New NormTune method boosts PEFT performance on reasoning tasks
Researchers have developed NormTune, a novel parameter-efficient fine-tuning (PEFT) strategy designed to improve performance on reasoning-intensive tasks. NormTune addresses the observed performance gap between PEFT and…
-
Qwen2.5 Coder Instruct 32B benchmark scores revealed in independent tests
Independent evaluations of the Qwen2.5 Coder Instruct 32B model have yielded specific benchmark scores. The model achieved 41.7% on GPQA, 63.5% on MMLU-Pro, 3.5% on Humanity's Last Exam, and 29.5% on LiveCodeBench. Thes…
-
DeepSeek open-sources production inference stack, DeepInfer
DeepSeek has open-sourced its production inference stack, DeepInfer, under an Apache 2.0 license. This toolkit is designed to power large-scale services, specifically for their DeepSeek-V4.1-Flash model, and offers supp…
-
SkillEval framework offers detailed LLM ability profiles beyond scalar scores
A new framework called SkillEval has been developed to provide a more nuanced evaluation of large language models (LLMs) beyond single scalar scores. SkillEval leverages LLMs' meta-cognitive abilities to identify interp…
-
K2-V2 and DeepSeek V3.2 benchmark results revealed · 2 sources tracked
Independent benchmarks reveal performance metrics for two large language models. K2-V2 (Medium) achieved scores of 59.8% on GPQA and 76.1% on MMLU-Pro, with a 28% score for long context reasoning. DeepSeek V3.2 (Reasoni…
-
Tri-21B-Think model achieves strong benchmark performance
The Tri-21B-Think model, a 21 billion parameter language model, has achieved notable performance on independent benchmarks. It scored 60.1% on the GPQA benchmark and 54.6% on IFBench. These results indicate that the mod…
-
Molmo 7B-D benchmark results show mixed performance
The Molmo 7B-D language model has been evaluated on several benchmarks, showing specific scores on GPQA, MMLU-Pro, and Humanity's Last Exam. The model also scored 0% on long-context reasoning tasks. These results were m…