cross entropy
PulseAugur coverage of cross entropy — every cluster mentioning cross entropy across labs, papers, and developer communities, ranked by signal.
9 day(s) with sentiment data
-
New adversarial training method targets model confidence structure
Researchers have developed a novel approach to adversarial training that addresses the issue of "pseudo-attacks" by focusing on the deformation of a model's confidence structure. This method integrates cross-entropy and…
-
New theory reformulates neural collapse for imbalanced and open-set data
A new theoretical framework for neural collapse (NC) has been developed, addressing challenges posed by imbalanced datasets and open-set scenarios. The research reformulates NC using class-wise averaged losses instead o…
-
New TVP loss function enhances speculative decoding for faster LLM inference
Researchers have developed TVP, a novel loss function family designed to improve speculative decoding in large language models. This method aims to accelerate inference by optimizing the acceptance rate of proposed toke…
-
Gated Knowledge Distillation Fails to Improve Low-Resource Bangla Generation
A new research paper explores gated knowledge distillation techniques for low-resource Bangla language generation, finding that these methods do not significantly improve upon standard knowledge distillation or cross-en…
-
AI model detects illegal wildlife trade products with 91% accuracy
Researchers have developed a deep-learning object detection framework to identify products derived from threatened species like elephants, pangolins, and tigers in online marketplaces. The system, trained using a datase…
-
New paper reveals numerical issues in ML libraries, proposes fixes
A recent paper by Barron et. al. identified numerical issues in the softmax and cross-entropy functions used across common machine learning libraries like PyTorch and JAX. Researchers demonstrated that property-based te…
-
Researchers Fix Numerical Bugs in Softmax and Cross-Entropy Implementations
Researchers have identified and fixed two numerical bugs in standard implementations of softmax, log-softmax, and cross-entropy. The first bug, Softmax Collapse, causes the target logit's gradient to round to zero when …
-
Draft-training objectives show diminishing returns in speculative decoding
A new research paper from arXiv explores the effectiveness of draft-training objectives in speculative decoding for large language models. The study found that while these objectives initially show gains over standard c…
-
New research highlights loss function misalignment in AI robustness evaluations
Researchers have identified a misalignment between surrogate loss functions and the zero-one loss in Wasserstein distributionally robust optimization (WDRO). This misalignment can lead to hypotheses with perfect robust …
-
New loss function improves AI set-valued classifiers on imbalanced data
Researchers have developed a modified version of the odd-K-out (OKO) loss function to improve the performance of set-valued classifiers on imbalanced datasets. This adapted OKO loss function enhances sample efficiency a…
-
New BOLT loss function aims to improve classifier training accuracy
Researchers have introduced BOLT, a novel training loss function designed to provide a universal upper bound on the Bayes error rate for classifiers. This new loss function, derived from the Bayes error rate, aims to gu…
-
New research explores Corrective Power in pairwise AUC optimization
A new research paper explores the concept of "Corrective Power" in optimizing pairwise AUC (Area Under the Curve) for machine learning models, particularly in scenarios with imbalanced classes. The study identifies that…
-
New Research Explores Implicit Bias in Hyperbolic Representation Learning
This paper delves into the implicit bias of hyperbolic representation learning for multiclass data, analyzing it through the lens of Busemann risk. The research decomposes the distance to class prototypes in hyperbolic …
-
AI research explores new methods for constraint-aware model training · 2 sources tracked
Two new research papers explore methods for training AI models to adhere to constraints more effectively. The first paper introduces Counterfactual KKT (CF-KKT), a framework that uses learned dynamics and optimal demons…
-
New research enhances Mixture-of-Experts LLMs with improved routing and diffusion models · 2 sources tracked
Two new research papers explore enhancements to Mixture-of-Experts (MoE) large language models. The first paper introduces token-error supervision to guide MoE routing, improving accuracy on benchmarks like Granite and …
-
New Entropy Regularization Method Improves AI Model Training for Verifiable Tasks
Researchers have identified a misalignment between standard cross-entropy (CE) training and the objective of producing correct outputs in verifiable domains like mathematical reasoning and code generation. This issue ar…
-
Adam optimizer stability phase diagram unveiled by researchers
Researchers have identified a stability phase diagram for the Adam optimizer, revealing how its two momentum timescales govern training instabilities. They discovered an approximately linear boundary in the $(\beta_1,\b…
-
New method identifies critical image regions for VQA models
Researchers have developed a new method called Counterfactual Search for Grounding Regions (CSGR) to identify image regions crucial for visual question answering (VQA) models. This approach intervenes in image regions t…
-
New loss function tackles 'center-class hedging' in ordinal classification
Researchers have introduced a new loss function called Adaptive Margin Ordinal Loss (AMOL) to address a specific failure mode in ordinal classification tasks, known as center-class hedging. This phenomenon causes neural…
-
New TPT Method Improves AI Model Calibration and Accuracy
Researchers have identified a limitation in test-time prompt tuning (TPT) methods that rely on entropy minimization, noting that these approaches can lead to overconfident predictions and degraded model calibration. To …