HellaSwag
PulseAugur coverage of HellaSwag — every cluster mentioning HellaSwag across labs, papers, and developer communities, ranked by signal.
3 day(s) with sentiment data
-
LLM benchmark contamination inflates scores but rarely reorders leaderboards
A new paper from arXiv investigates benchmark contamination in large language models, distinguishing between score inflation and leaderboard reordering. The research found that while contamination does inflate absolute …
-
New ROSI technique amplifies LLM safety alignment without fine-tuning
Researchers have developed a new technique called Rank-One Safety Injection (ROSI) to enhance the safety alignment of Large Language Models (LLMs). ROSI is a fine-tuning-free method that permanently steers a model's int…
-
VIDRAFT launches small AI model evaluation framework, hits Hugging Face top 10
VIDRAFT (지니젠AI), a Korean AI startup, has launched a specialized evaluation framework for small language models (SLMs). This framework, along with its associated dataset, has simultaneously achieved a top-10 ranking on …
-
LLM Leaderboards Skewed by Config-Fragile Items, Study Finds
A new paper from arXiv reveals that modern large language model (LLM) leaderboards are significantly influenced by "config-fragile" items, meaning the way questions are presented and answers are evaluated can drasticall…
-
New VA-DPO method enables controllable emotion generation in language models
Researchers have developed a new method called VA-DPO to enable language models to generate text with controllable emotions. Unlike previous methods that use discrete labels, VA-DPO specifies desired affect as a continu…
-
LLM evaluation metrics 'acc' vs 'acc_norm' explained
A technical discussion highlights the 'acc vs acc_norm' problem in evaluating Large Language Models (LLMs), particularly in multiple-choice tasks. The standard accuracy metric (acc) favors shorter answers due to its rel…
-
New Daedalus-150M model achieves faster CPU inference with hybrid architecture
Researchers have developed Daedalus-150M, a novel language model optimized for efficient CPU inference. This hybrid model combines sparse attention with short convolutions, allowing two-thirds of its architecture to avo…
-
New FPO method adapts LLMs without backward pass, boosting throughput
Researchers have developed a new method called Forward-Pass-Only (FPO) training that adapts large language models without requiring a backward pass through the model's layers. This technique achieves significantly highe…
-
Spectral outliers in Transformer attention reveal dominant learned structures
Researchers have applied Marchenko-Pastur random matrix theory to analyze pre-trained transformer attention weights, identifying spectral outliers that represent dominant learned structures. By zeroing these identified …
-
LLM sampling variation doesn't reveal model ignorance, study finds
A new research paper published on arXiv explores the limitations of stochastic sampling in large language models (LLMs). The study, titled "Stochastic Sampling is Epistemically Shallow: The Dimensionality Gap Between Te…
-
New Benchmark Suite Evaluates LLMs on Kyrgyz Language Understanding
Researchers have developed KyrgyzLLM-Bench, a new benchmark suite designed to evaluate large language models (LLMs) on the Kyrgyz language. This suite includes natively authored datasets like KyrgyzMMLU and KyrgyzRC, al…
-
Building Production-Grade LLM Evaluation Pipelines: From Vibes to Metrics · 8 sources tracked
This series of articles details the creation of production-grade evaluation pipelines for Large Language Models (LLMs), moving beyond subjective "vibe checks" to implement automated metrics. The authors emphasize the ne…
-
Prism Transformer introduces progressive head schedules for hierarchical attention
Researchers have introduced the Prism Transformer, a novel architecture that modifies the standard multi-head attention mechanism. Instead of allocating equal dimensional space to each attention head at every layer, Pri…
-
New methods boost LLM inference speed with adaptive decoding strategies
Researchers have developed BlockPilot, a novel approach to speculative decoding that adaptively predicts optimal block sizes for generating text. This method improves efficiency by learning a policy that selects block s…
-
General LLMs now outperform specialized clinical AI on benchmarks, but safety concerns persist
General-purpose large language models are now achieving performance levels comparable to or exceeding specialized clinical AI systems on various benchmarks, including those for structured knowledge and reasoning. For in…
-
New QAT Method Achieves Near-Lossless LLM Performance
Researchers have developed a new method for quantization-aware training (QAT) of large language models (LLMs) called Max-Window Scale Estimation. This technique addresses two failure modes: amax saturation, where delaye…
-
New QUIET benchmark objectively measures LLM creative writing
Researchers have introduced QUIET, a new benchmark designed to evaluate the creative generation capabilities of large language models. Unlike existing benchmarks that rely on multiple-choice formats or subjective human …
-
LLM benchmark costs analyzed: $0.12 for 3 tasks
Benchmarking three large language model tasks (GSM8K, HellaSwag, and TruthfulQA) on a single T4 GPU costs approximately $0.12. The analysis reveals that generative tasks are the primary cost driver, while log-likelihood…
-
Evaluate LLMs for under $1 using Qwen2.5-0.5B
This post details a cost-effective method for evaluating large language models, demonstrating that comprehensive benchmarks can be run for under a dollar. The author used a free Google Colab T4 instance to test the Qwen…
-
Aurora optimizer boosts neural network training efficiency
Researchers have introduced Aurora, a new optimizer designed to improve the training of large neural networks, particularly those with rectangular matrices. Aurora addresses issues like neuron death in MLP layers that c…