This paper investigates how the neural tangent kernel (NTK) behaves in convolutional neural networks (CNNs) when both depth and width are scaled together. The researchers found that NTK dispersion, a measure of the kernel's variability, increases with the ratio of depth to width (L/C) in specific initialization regimes. Furthermore, kernel rotation, which occurs during training, also generally increases with L/C, even when initial dispersion is low. These findings suggest that the L/C ratio plays a significant role in how CNNs deviate from the theoretical infinite-width limit, with distinct scaling behaviors for initialization and training stability. AI
IMPACT Understanding how depth-width scaling affects NTK behavior in CNNs could inform future architectural designs for improved training stability and performance.
RANK_REASON The item is a research paper detailing numerical investigations into the behavior of neural networks. [lever_c_demoted from research: ic=1 ai=1.0]
Read on Hugging Face Daily Papers →
- convolutional neural network
- Fully Connected Networks on a Diet With the Mediterranean Matrix Multiplication
- Hugging Face
- kernel rotation
- Neural tangent kernel
- NTK dispersion
- Recurrent Neural Networks
- Residual Networks Behave Like Ensembles of Relatively Shallow Networks
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →