A new research paper explores the performance gap between trained neural networks and their Neural Tangent Kernel (NTK) limits, particularly for tasks with compositional structure. The study introduces a dichotomy between Fourier complexity, which governs NTK kernel regression, and architectural complexity, which relates to the learning capabilities of deep ReLU networks. The findings indicate that NTK estimators can be exponentially suboptimal compared to standard networks when these complexities diverge, as demonstrated on specific models like the iterated sawtooth and hypercube sparse-parity model. AI
IMPACT Highlights a fundamental gap in understanding neural network learning dynamics, suggesting architectural choices significantly impact performance beyond kernel methods.
RANK_REASON The cluster contains an academic paper detailing theoretical findings in machine learning.
- Fourier complexity
- hypercube sparse-parity model
- iterated sawtooth
- Minimax Rate of Testing in Sparse Linear Regression
- Neural tangent kernel
- rectifier
- Two-layer networked learning control using self-learning fuzzy control algorithms
AI-generated summary · Google Gemini · from 2 sources. How we write summaries →
