rectifier
PulseAugur coverage of rectifier — every cluster mentioning rectifier across labs, papers, and developer communities, ranked by signal.
No coverage in the last 90 days.
-
AI research links data geometry to neural network generalization
This paper theoretically investigates how data geometry influences generalization in overparameterized neural networks trained below the edge of stability. It derives generalization bounds for two-layer ReLU networks, s…
-
Researchers explore how gradient descent adapts neural network capacity to tasks
Researchers have developed a theoretical framework to explain how neural networks adapt their capacity to specific tasks during gradient descent training. The study identifies three key dynamical principles—mutual align…
-
New lattice-based framework for piecewise GLMs inspired by renormalization group theory
Researchers have introduced a novel framework for generalized linear models inspired by renormalization group theory. This approach utilizes additive hierarchical expansions to create models that are locally linear, sim…
-
New research explores active learning for conditional generative compressed sensing
Researchers have developed a new framework for conditional generative compressed sensing, specifically for image recovery from subsampled Fourier measurements using prompt-conditioned generative models. This approach di…
-
Photonic ROM architecture enables high-speed, reconfigurable lookup tables for accelerators
Researchers have developed OptiLookUp, a novel photonic architecture that utilizes integrated microring resonators to create a reconfigurable optical read-only memory (ROM). This system encodes input-output mappings dir…
-
New mechanistic estimation method outperforms sampling for wide random MLPs
Researchers have developed a new method for estimating the expected output of wide, randomly initialized multilayer perceptrons (MLPs) without needing to run samples through the model. This "mechanistic estimation" appr…
-
Researchers develop exact ReLU realization for tensor-product refinement iterates
Two new arXiv papers explore advanced mathematical techniques for realizing ReLU (Rectified Linear Unit) functions in neural networks. The first paper, "Exact ReLU realization of tensor-product refinement iterates," ext…
-
Neural networks achieve super-fast convergence and represent complex functions with floating-point arithmetic
Two new arXiv papers explore theoretical aspects of neural network convergence and representation capabilities. The first paper demonstrates that neural network classifiers can achieve super-fast convergence rates under…
-
Deep neural networks provably overcome curse of dimensionality for PDEs
Researchers have demonstrated that deep neural networks (DNNs) can overcome the curse of dimensionality when approximating solutions to Kolmogorov partial differential equations. This mathematical proof extends previous…
-
Researchers evolve activation functions to handle missing data in neural networks
Researchers have developed a novel approach called Three-Channel Evolved Activations (3C-EA) to address challenges in machine learning when dealing with missing data. Unlike traditional activation functions, 3C-EA incor…
-
New research explores activation functions beyond ReLU in neural networks
A new paper explores the theoretical underpinnings of neural network kernels, specifically focusing on activation functions beyond the standard ReLU. Researchers characterized the Reproducing Kernel Hilbert Spaces (RKHS…
-
Ternary neural networks offer theoretical expressivity comparable to standard NNs
Researchers have theoretically analyzed the expressivity of ternary neural networks, which use parameters restricted to {-1, 0, +1}. The study focuses on regression networks with ReLU activation functions, proving that …
-
MLP skip connections can't be absorbed into residual-free models
Researchers have investigated whether a skip connection around a single-hidden-layer MLP can be absorbed into a residual-free MLP of the same width. They found that for certain activation functions like ReLU^2 and ReGLU…
-
Self-supervised networks create fewer linear regions for comparable accuracy
A new study published on arXiv investigates the complexity of linear regions within self-supervised deep ReLU networks. Researchers found that self-supervised learning methods create fewer linear regions compared to sup…
-
Beyond Linearity in Attention Projections: The Case for Nonlinear Queries
Researchers are exploring the fundamental mechanisms behind transformer attention, with new papers analyzing its gradient flow structure and dynamics. One study interprets attention as a gradient flow on a unit sphere, …
-
Researchers present new integral representations and bounds for two-layer ReLU networks
Researchers have developed a novel method for constructing explicit integral representations of two-layer ReLU networks, enabling simpler representations for multivariate polynomials. This approach yields quantitative b…
-
Researchers develop new training methods for neural networks to improve MILP tractability
Researchers have developed new training regularizers for neural network surrogate models that directly improve their tractability within mixed-integer linear programs (MILPs). These regularizers penalize factors like bi…
-
SOC-ICNN: From Polyhedral to Conic Geometry for Learning Convex Surrogate Functions
Researchers have introduced SOC-ICNN, a novel neural network architecture that expands the representational capabilities beyond classical ReLU-based Input Convex Neural Networks (ICNNs). By generalizing from Linear Prog…
-
New GEM activation functions offer smoother, rational alternatives to ReLU
Researchers have introduced Geometric Monomial (GEM), a new family of activation functions designed for deep neural networks. These functions utilize purely rational arithmetic and offer $C^{2N}$-smoothness, aiming to i…