Z-Plane Neural Networks: Bounded Geometric Activation Replaces ReLU and LayerNorm
Researchers have introduced a novel neural network architecture called the Z-Plane Neural Network, which replaces traditional activation functions like ReLU and normalization techniques like LayerNorm. This new approach maps hidden states into 2D phasor bundles on a hypersphere, utilizing a geometric activation function called Radial Bounding. This method aims to prevent gradient instability, avoid dead neurons, and preserve directional information. A 100-layer Z-Plane Multi-Layer Perceptron demonstrated successful convergence and numerical stability on the MNIST dataset, achieving 98.34% accuracy without ReLU or LayerNorm, suggesting that bounded geometric activation alone is sufficient for deep learning. AI
IMPACT Introduces a novel activation and normalization method that could improve stability and information preservation in deep learning models.