knowledge distillation
PulseAugur coverage of knowledge distillation — every cluster mentioning knowledge distillation across labs, papers, and developer communities, ranked by signal.
- uses large-language models 90%
- instance of On-Policy Distillation 90%
- used by large-language models 90%
- used by distilled AI model 90%
- developed large-language models 70%
- instance of CIFAR-100 70%
- instance of DagsHub 70%
- used by alphaXiv 70%
- used by Gotit.pub 70%
- competes with On-Policy Distillation 70%
- developed by large-language models 70%
- used by CIFAR-100 70%
13 day(s) with sentiment data
-
Gated Knowledge Distillation Fails to Improve Low-Resource Bangla Generation
A new research paper explores gated knowledge distillation techniques for low-resource Bangla language generation, finding that these methods do not significantly improve upon standard knowledge distillation or cross-en…
-
AI advances quantum error correction code design and decoding
Researchers have developed novel AI approaches to improve the design and decoding of quantum error-correcting codes. One method, GFlowQEC, utilizes generative flow networks to efficiently search for and propose high-sco…
-
AI research focuses on enabling models to forget information intentionally
Researchers are exploring methods for artificial intelligence systems to intentionally forget information, a capability that could enhance privacy and security. This is particularly relevant for AI models that continuou…
-
AWS enhances ML with new tools for Bedrock and SageMaker
AWS is enhancing its machine learning capabilities with new tools and techniques. The company is introducing Crawl4AI to help prepare web content for Amazon Bedrock Knowledge Bases. Additionally, AWS is enabling the dev…
-
History Forcing enhances long-horizon video generation consistency
Researchers have developed a new technique called History Forcing to improve the long-range consistency of autoregressive video generation models. This method uses historical context residuals to correct distillation ta…
-
Steering vectors show promise as LLM training signal
Researchers have explored the use of steering vectors (SVs) as a training signal for post-training large language models (LLMs). Their framework investigates SV injection during supervised learning, including supervised…
-
AI reliability-safety trade-offs explored via information theory and distillation
Researchers have established an information-theoretic bound on the trade-off between reliability and safety in AI systems, particularly as it evolves through knowledge distillation. They developed a Boltzmann-machine mo…
-
AI model distillation economics explored in new arXiv paper
A new paper from arXiv explores the economic implications of model distillation in the generative AI landscape. The research analyzes how smaller firms can leverage data from larger, closed-weight models to train their …
-
AI distillation creates faster MLIPs for battery electrolyte simulation
Researchers have developed a knowledge distillation technique to create faster, more accurate machine learning interatomic potentials (MLIPs) for simulating liquid electrolytes. These "student" models, distilled from la…
-
New technique boosts performance of compressed transformer models
Researchers have developed a novel post-hoc embedding alignment technique to improve the performance of compressed transformer models. This method, called the Alignment Adapter (AlAd), uses a small feed-forward network …
-
New PCoreSet strategy enhances active learning via VLM knowledge distillation
Researchers have introduced PCoreSet, a novel active learning strategy that leverages knowledge distillation from vision-language models (VLMs). This approach, termed ActiveKD, addresses the challenge of limited labeled…
-
New Dual-Head Optimization boosts VLM knowledge transfer for semi-supervised learning
Researchers have developed Dual-Head Optimization (DHO), a novel technique to improve semi-supervised learning by effectively transferring knowledge from powerful vision-language models (VLMs) to smaller, task-specific …
-
New framework unifies LLM training algorithms via distribution matching
Researchers have introduced a new framework for understanding large language model training algorithms, viewing them through the lens of distribution matching. This perspective suggests that various training methods can…
-
New DINO augmentation strategy boosts glaucoma classification accuracy
Researchers have developed a new augmentation strategy for the DINO self-supervised learning framework specifically tailored for medical imaging, particularly for glaucoma classification using retinal fundus images. The…
-
AI model compression for edge devices detailed in new research
A new research paper explores model compression techniques for deploying AI on heterogeneous edge platforms, focusing on predictive maintenance for rotating machinery. The study evaluated four state-of-the-art compressi…
-
New GIFT method enhances LLM reasoning via gradient filtering
Researchers have introduced GIFT, a novel method for training large language models that improves complex reasoning abilities. GIFT addresses the conflict between reinforcement learning (RL) and knowledge distillation b…
-
New method enhances split computing with federated learning and personalization
Researchers have developed a new method for training head networks in split computing environments using federated learning and knowledge distillation. This approach addresses challenges related to limited labeled data …
-
New KETCHUP method improves reinforcement learning for text generation
Researchers have introduced KETCHUP, a new method for knowledge distillation in text generation tasks using reinforcement learning. KETCHUP employs a k-step return estimation technique based on the Bellman Optimality Eq…
-
New distillation method transfers AI model safety across families
Researchers have introduced a novel method for transferring "native safety"—robust safety acquired during pretraining—from a specialized teacher model to a standard-pretrained student model. This on-policy distillation …
-
New DAR method embeds persistent backdoors in models using clean data
Researchers have developed a new method called DAR (Discover-and-Relabel) that can embed persistent backdoors into machine learning models, even when using clean data for training. Unlike traditional backdoor attacks th…