Optimized Federated Knowledge Distillation with Distributed Neural Architecture Search
Researchers have developed FedKDNAS, a novel federated learning framework that optimizes model selection and knowledge distillation for heterogeneous client devices. This approach allows each client to autonomously choose a lightweight model tailored to its specific accuracy and resource constraints. The framework then uses a hybrid objective for training, incorporating both supervised learning and knowledge distillation, and shares only predictions on a public reference set. Evaluations show FedKDNAS significantly improves accuracy under non-IID conditions, reduces CPU usage, and drastically cuts communication overhead compared to existing baselines. AI
IMPACT Enhances federated learning efficiency and accuracy on heterogeneous devices, potentially accelerating collaborative AI development.