cross entropy
PulseAugur coverage of cross entropy — every cluster mentioning cross entropy across labs, papers, and developer communities, ranked by signal.
5 day(s) with sentiment data
-
Loss functions explained: MSE, MAE, Huber, and cross-entropy
The article explains the dual role of loss functions in machine learning: quantifying errors and guiding model training through their derivatives. It details how Mean Squared Error (MSE) converges to the mean and Mean A…
-
New adaptive gradient descent method improves ML optimization
Researchers have developed a new adaptive gradient descent method that improves optimization for machine learning models by focusing on the descent direction rather than the full gradient variation. This approach, detai…
-
Small-population ES fine-tuning for LLMs shows promise with reward adjustments
A new research paper explores the effectiveness of Evolutionary Strategies (ES) for fine-tuning large language models, particularly when using binary rewards. The study found that the perceived need for large population…
-
New loss function improves graph neural networks for recommendations
Researchers have developed a new method called Cardinality-Decomposed Loss (CDL) to improve the performance of graph neural networks in recommendation systems. Traditional methods often use a single loss function like B…
-
New research tackles 3D point cloud segmentation challenges
Two new research papers explore advanced techniques for 3D point cloud segmentation and understanding. The first paper investigates the effectiveness of standard cross-entropy loss in handling class imbalance in 3D poin…
-
Dice Loss Proposed for Data-Imbalanced NLP Tasks
A research paper proposes using Dice loss as an alternative to standard cross-entropy for natural language processing tasks that suffer from severe data imbalance. This approach, based on the Sorensen-Dice coefficient o…
-
New CoCo loss function enhances embedding structure and convergence
Researchers have developed a new loss function called CoCo, designed to create normalized and well-structured data representations. CoCo encourages classes to collapse internally while contrasting with other classes, ai…
-
New CoCo loss function enhances embedding quality and training speed
Researchers have introduced CoCo, a novel loss function designed to create normalized and well-structured data representations. This function promotes intra-class collapse and inter-class contrast, enabling neural netwo…
-
Research identifies key factors for effective representational priors in AI model generalization
A new research paper explores the factors that make representational priors effective in machine learning, particularly in the context of "grokking," where models transition from memorization to generalization. The stud…
-
Research links AI grokking delay to representational structure formation
Researchers have investigated the phenomenon of grokking, where a model generalizes long after its training data has been fully memorized. Through experiments with a one-layer transformer, they causally demonstrated tha…
-
New LP-SFT method preserves language model capabilities during fine-tuning
Researchers have introduced LP-SFT, a novel supervised fine-tuning method designed to preserve the inherent entropy structure of pretrained language models. Standard fine-tuning can degrade existing capabilities by over…
-
New SMMD training method enhances numerical accuracy in LLMs
Researchers have developed a new training objective called Smooth Maximum Mean Discrepancy (SMMD) to improve the numerical precision of large language models (LLMs). Standard cross-entropy training treats numerical toke…
-
New Ordinal Cross-Entropy framework enhances deep learning for medical predictions
Researchers have introduced a new framework called Ordinal Cross-Entropy (OCE) designed to improve the accuracy of deep neural networks in medical applications where target labels have an inherent ordinal structure. Tra…
-
Mean Field Control Analysis of Transformer Layers under Cross-Entropy Training
Researchers have analyzed Transformer layers within a cross-entropy training framework using a continuous-depth mean field control perspective. They treat depth as time and layer parameters as controls, modeling the Tra…
-
New research probes Transformer energy use, learned linearity, and training dynamics
Recent research explores the intricacies of Transformer models, focusing on their energy consumption, internal linear properties, and training dynamics. One paper introduces a scaling model to predict energy usage durin…
-
Weight norm's role in neural network grokking clarified
Researchers have investigated the phenomenon of 'grokking' in neural networks, where a model transitions from memorization to generalization. Their findings indicate that the weight norm, previously thought to be the pr…
-
New losses achieve Neural Collapse faster in supervised learning
Researchers have introduced new methods, NTCE and NONL, to improve supervised classification by achieving Neural Collapse (NC) more efficiently. These techniques address limitations in existing paradigms like cross-entr…
-
Researchers develop Evolutionary Dynamic Loss for distribution-free pretraining
Researchers have developed a new framework called Evolutionary Dynamic Loss (EDL) for pretraining classification losses. EDL learns a transferable loss function using synthetic data, avoiding the need for real samples d…
-
Linear-Core Surrogates offer smooth loss functions with linear rates for classification
Researchers have introduced Linear-Core (LC) Surrogates, a novel family of convex loss functions designed to combine the benefits of smooth and piecewise-linear losses in machine learning. These surrogates are different…
-
Contrastive learning advances model robustness and transparency in AI
Contrastive learning is a machine learning technique that creates an embedding space where similar data points are grouped together and dissimilar ones are separated. This method can be applied in both supervised and un…