Activation bias in neural networks impacts learning dynamics, research finds

A new research paper published on arXiv explores the concept of "activation bias" in neural networks, a phenomenon where common activation functions like ReLU, GELU, Swish, and Sigmoid predominantly produce positive outputs. The study demonstrates that while this bias can initially speed up loss reduction, it leads to less stable learning trajectories and a steeper loss landscape. Reducing activation bias, conversely, slows initial learning but results in better final performance and improved late-stage results in larger vision and language models. The findings suggest that activation bias is a significant factor that should be considered in the design of future activation functions. AI

IMPACT This research could lead to more stable and performant neural network training by informing the design of new activation functions.

RANK_REASON The cluster contains a research paper detailing novel findings about neural network activation functions. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• alphaXiv
• arXiv
• CatalyzeX
• DagsHub
• Gelu
• Gotit.pub
• Hugging Face
• IArxiv
• rectifier
• ScienceCast
• sigmoid function
• Swish

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →