HellaSwag
PulseAugur coverage of HellaSwag — every cluster mentioning HellaSwag across labs, papers, and developer communities, ranked by signal.
4 day(s) with sentiment data
-
Spectral outliers in Transformer attention reveal dominant learned structures
Researchers have applied Marchenko-Pastur random matrix theory to analyze pre-trained transformer attention weights, identifying spectral outliers that represent dominant learned structures. By zeroing these identified …
-
LLM sampling variation doesn't reveal model ignorance, study finds
A new research paper published on arXiv explores the limitations of stochastic sampling in large language models (LLMs). The study, titled "Stochastic Sampling is Epistemically Shallow: The Dimensionality Gap Between Te…
-
New Benchmark Suite Evaluates LLMs on Kyrgyz Language Understanding
Researchers have developed KyrgyzLLM-Bench, a new benchmark suite designed to evaluate large language models (LLMs) on the Kyrgyz language. This suite includes natively authored datasets like KyrgyzMMLU and KyrgyzRC, al…
-
Building Production-Grade LLM Evaluation Pipelines: From Vibes to Metrics · 8 sources tracked
This series of articles details the creation of production-grade evaluation pipelines for Large Language Models (LLMs), moving beyond subjective "vibe checks" to implement automated metrics. The authors emphasize the ne…
-
Prism Transformer introduces progressive head schedules for hierarchical attention
Researchers have introduced the Prism Transformer, a novel architecture that modifies the standard multi-head attention mechanism. Instead of allocating equal dimensional space to each attention head at every layer, Pri…
-
New methods boost LLM inference speed with adaptive decoding strategies
Researchers have developed BlockPilot, a novel approach to speculative decoding that adaptively predicts optimal block sizes for generating text. This method improves efficiency by learning a policy that selects block s…
-
General LLMs now outperform specialized clinical AI on benchmarks, but safety concerns persist
General-purpose large language models are now achieving performance levels comparable to or exceeding specialized clinical AI systems on various benchmarks, including those for structured knowledge and reasoning. For in…
-
New QAT Method Achieves Near-Lossless LLM Performance
Researchers have developed a new method for quantization-aware training (QAT) of large language models (LLMs) called Max-Window Scale Estimation. This technique addresses two failure modes: amax saturation, where delaye…
-
New QUIET benchmark objectively measures LLM creative writing
Researchers have introduced QUIET, a new benchmark designed to evaluate the creative generation capabilities of large language models. Unlike existing benchmarks that rely on multiple-choice formats or subjective human …
-
LLM benchmark costs analyzed: $0.12 for 3 tasks
Benchmarking three large language model tasks (GSM8K, HellaSwag, and TruthfulQA) on a single T4 GPU costs approximately $0.12. The analysis reveals that generative tasks are the primary cost driver, while log-likelihood…
-
Evaluate LLMs for under $1 using Qwen2.5-0.5B
This post details a cost-effective method for evaluating large language models, demonstrating that comprehensive benchmarks can be run for under a dollar. The author used a free Google Colab T4 instance to test the Qwen…
-
Aurora optimizer boosts neural network training efficiency
Researchers have introduced Aurora, a new optimizer designed to improve the training of large neural networks, particularly those with rectangular matrices. Aurora addresses issues like neuron death in MLP layers that c…