Automatically Differentiable Nonlinear Tensor Networks (ADNTNs) for Exponential Compression of Deep Neural Networks
Two new research papers explore novel methods for compressing deep neural networks. The first paper introduces Automatically Differentiable Nonlinear Tensor Networks (ADNTNs), which use a hierarchy of small core tensors and nonlinear activations to achieve significant compression ratios, sometimes exceeding 77,000x, while maintaining or even improving accuracy on benchmarks like AlexNet and VGG-16. The second paper proposes a complementary approach using Approximate Differential Equivalence to aggregate neurons with similar functional behavior, encoding networks as polynomial ODE systems and identifying neurons with matching dynamics. This method offers a principled alternative to weight-centric pruning, achieving substantial parameter reduction with a smooth trade-off between model size and accuracy. AI
IMPACT These novel compression techniques could lead to significantly smaller and more efficient AI models, enabling deployment on resource-constrained devices.