Scaling Laws and Tradeoffs in Recurrent Networks of Expressive Neurons
Researchers have introduced the Expressive Leaky Memory (ELM) Network, a novel recurrent neural network architecture designed to better mimic the functional components of cortical neurons. This new model allows for independent tuning of the number of units, per-unit complexity, and connectivity, addressing a key difference from mainstream ML models that use simpler units. Experiments on sequence benchmarks like the SHD-Adding task and Enwik8 language modeling demonstrated that performance improves with increased complexity, width, and connectivity, and a theoretical framework was developed to explain these scaling laws and tradeoffs. AI
IMPACT Introduces a new neural network architecture that could lead to more biologically plausible and potentially more efficient AI models for sequence processing.