knowledge distillation
PulseAugur coverage of knowledge distillation — every cluster mentioning knowledge distillation across labs, papers, and developer communities, ranked by signal.
13 day(s) with sentiment data
-
Model distillation attacks pose growing AI security threat
Model distillation attacks, where a smaller model learns from a larger one's outputs, pose an under-recognized security threat to AI systems. These attacks can bypass safety alignments, leading to models that generate h…
-
New IF-Beta framework streamlines knowledge distillation with data pruning
Researchers have developed IF-Beta, a novel framework for efficient knowledge distillation that utilizes learnable data pruning. This method combines influence functions with a Beta distribution-parameterized sampling p…
-
Knowledge distillation outperforms SFT in low-data LLM training
A new paper explores knowledge distillation (KD) for post-training large language models (LLMs), finding it outperforms supervised fine-tuning (SFT) in low-data scenarios. The effectiveness of KD diminishes as more data…
-
Hugging Face paper reveals "subliminal learning" in LLMs, impacting auditability
A new paper from Hugging Face explores the concept of "subliminal learning" in language models, where a student model can inherit hidden traits from a teacher model through distillation data that doesn't explicitly name…
-
New TALAS framework improves language model distillation efficiency
Researchers have introduced TALAS, a novel framework for knowledge distillation in pre-trained language models. TALAS synchronizes hierarchical alignment with advanced optimization techniques to improve efficiency and p…
-
New LEAP curriculum boosts Vision Transformer distillation efficiency
Researchers from the University of Oxford have introduced LEAP, a novel training curriculum designed to improve the efficiency of knowledge distillation for Vision Transformers (ViTs). LEAP utilizes a progressive approa…
-
New knowledge distillation method boosts land-use image classification accuracy
Researchers have developed an improved knowledge distillation framework to compress deep convolutional neural networks for land-use image classification. This approach uses a teacher-student learning paradigm where a VG…
-
Mixup distillation enhances student model accuracy and calibration
Researchers have explored the interaction between Knowledge Distillation (KD) and mixup techniques in machine learning, particularly when mixup is applied only during the student model's training. They found that this s…
-
Soft prompt distillation enhances on-device LLM safety
Researchers have developed a new method for making large language models safer and more efficient for use on devices with limited resources. The technique involves using "soft prompts" combined with distillation to tran…
-
New research advances flow matching models with theoretical and algorithmic improvements
Researchers have developed new theoretical foundations and practical algorithms for flow matching models, a type of generative model. One paper establishes convergence guarantees for neural network-parameterized conditi…
-
New Confusion Distillation method enhances self-distillation in ML
Researchers have developed a new method called Confusion Distillation (CD) to improve self-distillation in machine learning models. This technique analyzes the feature learning process in student models, revealing that …
-
New framework unifies knowledge transfer analysis in ML
Researchers have developed a unified spectral analysis framework to understand knowledge transfer in machine learning, particularly in high-dimensional linear regression. This framework explains how knowledge distillati…
-
New KD method improves dead tree detection in diverse forests
Researchers have developed a new method for detecting dead trees in aerial imagery using knowledge distillation (KD) to improve model generalization across different forest types. The TreeMort-1T-UNet model, initially t…
-
NVIDIA's X-Token enables cross-tokenizer knowledge distillation for AI models
NVIDIA researchers have developed X-Token, a novel method for knowledge distillation that allows smaller AI models to learn from larger, incompatible teacher models. Unlike previous methods that struggle with different …
-
Student capacity and architecture correctness key to knowledge distillation
A new study published on arXiv investigates the effectiveness of knowledge distillation (KD) in ResNet models for image classification on CIFAR-10. The research found that a student model's capacity significantly impact…
-
LoopFM framework enhances foundation model knowledge transfer for recommendation systems
Researchers have developed LoopFM, a novel framework designed to improve knowledge transfer from large foundation models (FMs) to smaller vertical models (VMs). Unlike traditional knowledge distillation, LoopFM structur…
-
New BackWeak method implants stealthy backdoors in AI model distillation
Researchers have developed a new method called BackWeak to implant backdoors into knowledge distillation processes. This technique uses subtle, imperceptible triggers and simple fine-tuning of teacher models. The backdo…
-
Lightweight Bangla Medical Entity Recognition Framework Developed
Researchers have developed a new, lightweight framework for Bangla medical entity recognition designed for resource-constrained environments. The system utilizes a hybrid Transformer-CRF architecture, starting with a 12…
-
Research tackles knowledge distillation attacks with adaptive defenses
A research paper explores knowledge distillation attacks and defenses, proposing efficient methods to counter adaptive attacks. This work is particularly useful for teams focused on the security and robustness of distil…
-
New CIST technique enhances knowledge distillation with adaptive temperatures
Researchers have developed a new knowledge distillation technique called CIST, which addresses the limitations of fixed temperature scaling in transferring knowledge from teacher to student models. CIST assigns separate…