MMLU-Pro
PulseAugur coverage of MMLU-Pro — every cluster mentioning MMLU-Pro across labs, papers, and developer communities, ranked by signal.
- instance of VIDRAFT 90%
- instance of Darwin-180B-RSI 90%
- used by Qwen3.8 Flash-Next 90%
- instance of long-context reasoning 70%
- instance of SciCode 70%
- used by Math-500 70%
- instance of Math-500 70%
- instance of Gotit.pub 70%
- instance of Qwen3.5 4B 70%
- used by VIDRAFT 70%
- instance of IFBench 70%
- instance of alphaXiv 70%
19 day(s) with sentiment data
-
Tether AI Research releases BotPsy-0.8B agent model for on-device use
Tether AI Research has released BotPsy-0.8B, an open-source, on-device agent model designed for function calling. Built upon Qwen3.5-0.8B, BotPsy-0.8B demonstrates significant improvements in tool use and agentic benchm…
-
New LLM techniques: Byte-level conversion and efficient decoding strategies
Researchers have developed a method to convert a 27B parameter hybrid-attention LLM, Qwen3.8, into a pure byte-level model. This byte-level model, trained on self-generated data, shows competitive performance on several…
-
AI model evaluation evolves from token prediction to complex workflows
The evaluation of frontier AI models has evolved through distinct eras, moving from simple token prediction and log-likelihood grading to assessing instruction following, then to verifying math, code, factuality, and sa…
-
DRVEXO-4B Unified integrates vision, speech, and tool use in 4B model
DRVEXO-4B Unified is a new 4 billion parameter model that integrates image understanding, speech input and output, text conversation, and tool-use capabilities into a single package. It has demonstrated strong performan…
-
TwIL-8B claims top scores but faces scrutiny over benchmarks and licensing
webAI has released TwIL-8B, a fine-tuned version of IBM's Granite 4.2 8B model, which claims superior performance on coding and math benchmarks compared to Qwen3.5-9B. However, a closer examination of the model card rev…
-
AI models favor similar code, risking overlooked bugs in reviews
A new study suggests that AI models tend to favor and agree with outputs from similar models, potentially leading to overlooked errors in code reviews. Researchers found that AI judges often scored models more favorably…
-
WebAI releases TwIL-8B reasoning model, tops MATH-500 benchmark
WebAI has launched TwIL-8B, an 8.8 billion parameter reasoning model built upon IBM's Granite 4.2 8B. This model excels in mathematics and coding tasks, achieving a score of 98.2 on the MATH-500 benchmark, and is design…
-
DepthGraft architecture enhances LLM performance via hierarchical KV sharing
Researchers have introduced DepthGraft, a novel hierarchical KV-sharing architecture designed to improve the performance of large language models by addressing information dilution in deep layers. This method reconstruc…
-
125B model runs on 12GB gaming GPU via Strata fork
A developer has forked the Strata model-loading framework to enable running a 125-billion-parameter model, Qwen3.8 Flash-Next, on a single 12GB RTX 3060 gaming card. The modifications, detailed in a blog post, involve o…
-
DepthGraft architecture improves deep language model training
Researchers have introduced DepthGraft, a novel hierarchical cross-layer KV sharing architecture designed to enhance the training of deep language models. This method addresses the issue of information dilution in deep …
-
New research explores efficient test-time reasoning for LLMs · 10 sources tracked
Multiple research papers explore novel methods for enhancing the reasoning capabilities of large language models (LLMs) at test time. Techniques include learning when to apply teacher supervision to prevent "Teacher Dri…
-
LLM evaluation flawed by minor weight changes and truncation, study finds
A recent analysis revealed that even minor variations in model weights, as small as 0.005%, can lead to significant score differences of up to 1.4%p in LLM evaluations. The study found that repeating the same evaluation…
-
UkisAI releases Swift GLM-5.3 and Ornith 9B models with reduced token usage
UkisAI has released two new Swift models: Swift GLM-5.3-Flash and Swift Ornith 9B. The Swift GLM-5.3-Flash model achieves a 61.4% reduction in thinking tokens on the MMLU-PRO benchmark with a minimal accuracy loss of 0.…
-
New framework scales LLM evaluation using self-confidence and LLM judges
This paper introduces a statistical framework for evaluating large language models (LLMs) and multimodal large language models (MLLMs) using both direct human auditing and LLM-assisted methods. It proposes confidence-st…
-
AI evaluation harnesses studied for defects and effectiveness
A new preregistered study published on arXiv investigates the effectiveness of AI model evaluation harnesses, particularly focusing on visibility, baseline adequacy, and defects. The research utilized models ranging fro…
-
Study: Prompt position and authority influence LLM answers
A new study published on arXiv explores how to influence language models' answers, finding that the position of advice within a prompt significantly impacts its effectiveness. The research indicates that advice placed j…
-
New Fin-SD technique boosts open-weight models for financial reasoning
Researchers have developed Fin-SD, a novel self-distillation technique designed to enhance the multi-step reasoning capabilities of open-weight language models, particularly for financial applications. This method invol…
-
Qwen2.5 Coder Instruct 32B benchmark scores revealed in independent tests
Independent evaluations of the Qwen2.5 Coder Instruct 32B model have yielded specific benchmark scores. The model achieved 41.7% on GPQA, 63.5% on MMLU-Pro, 3.5% on Humanity's Last Exam, and 29.5% on LiveCodeBench. Thes…
-
Hugging Face benchmark leaders lack verification, score gaps minimal
A recent analysis of Hugging Face's official benchmarks revealed that none of the top-ranked models have undergone third-party verification. The study, conducted by VIDRAFT, examined 44 leaderboards and found that while…
-
ULME framework unifies LLM embedding and generation capabilities
Researchers have developed ULME, a novel post-training framework designed to unify text embedding and language generation within a single causal decoder-only LLM. Unlike previous models that treat these capabilities sep…