rectifier
PulseAugur coverage of rectifier — every cluster mentioning rectifier across labs, papers, and developer communities, ranked by signal.
4 天有情绪数据
-
超越注意力投影的线性：非线性查询的论证
研究人员正在探索 Transformer 注意力机制背后的基本原理，新论文分析了其梯度流结构和动态。一项研究将注意力解释为单位球面上的梯度流，识别影响多头设置中 token 聚类和稳定性的因素。另一篇论文研究了用于复杂性控制的关键训练窗口，确定 Transformer 何时优先考虑推理而非记忆。此外，研究还揭示了深度神经网络中几何连续性的起源，将其归因于残差连接和对称性破坏的非线性，并考察了“注意力汇聚”现象的结构原因。
-
Researchers present new integral representations and bounds for two-layer ReLU networks
Researchers have developed a novel method for constructing explicit integral representations of two-layer ReLU networks, enabling simpler representations for multivariate polynomials. This approach yields quantitative b…
-
Researchers develop new training methods for neural networks to improve MILP tractability
Researchers have developed new training regularizers for neural network surrogate models that directly improve their tractability within mixed-integer linear programs (MILPs). These regularizers penalize factors like bi…
-
SOC-ICNN: From Polyhedral to Conic Geometry for Learning Convex Surrogate Functions
Researchers have introduced SOC-ICNN, a novel neural network architecture that expands the representational capabilities beyond classical ReLU-based Input Convex Neural Networks (ICNNs). By generalizing from Linear Prog…
-
新的 GEM 激活函数提供了比 ReLU 更平滑、更具理性的替代方案
研究人员推出了一种名为 Geometric Monomial (GEM) 的新型激活函数族，专为深度神经网络设计。这些函数采用纯粹的有理数算术，并提供 $C^{2N}$-平滑性，旨在克服标准 ReLU 的局限性。实验表明，GEM 变体在 CIFAR-10、CIFAR-100、MNIST、GPT-2 和 BERT-small 等各种基准测试中，其性能可媲美甚至超越 GELU 等成熟函数。