Dataset Distillation Theory Explained for Two-Layer Neural Networks

Researchers have theoretically analyzed dataset distillation algorithms applied to gradient-based training of two-layer neural networks. The study focuses on a non-linear task structure called the multi-index model, proving that the low-dimensional structure of the problem is efficiently encoded into the resulting distilled data. This synthetic data can reproduce a model with high generalization ability, requiring a memory complexity of $\tilde{\Theta}$$(r^2d+L)$, where $d$ and $r$ are the input and intrinsic dimensions of the task. AI

IMPACT Provides theoretical underpinnings for dataset distillation, potentially improving efficiency in model training and data storage.

RANK_REASON Academic paper detailing theoretical analysis of dataset distillation algorithms. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv stat.ML →

• arXiv
• dataset distillation
• multi-index model
• two-layer neural networks
• Yuri Kinoshita

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →