Gelu
PulseAugur coverage of Gelu — every cluster mentioning Gelu across labs, papers, and developer communities, ranked by signal.
- instance of SwiGLU 90%
- instance of Swish 70%
- used by Softmax 70%
- used by multilayer perceptron 70%
- instance of sigmoid function 70%
- instance of hyperbolic tangent 70%
- competes with Swish 70%
- used by Swish 70%
- competes with Silu Activation Function 60%
- instance of Silu Activation Function 50%
- affiliated with Swish 50%
- used by SwiGLU 50%
1 day(s) with sentiment data
-
New method uses FFN curvature to recover hidden transformer structures
Researchers have developed a novel method for cryptanalyzing transformer feed-forward networks (FFNs) by examining their curvature. This technique exploits a second-order leakage channel, specifically projected input He…
-
New method extracts bias-free GLU blocks from language models
Researchers have developed a new method for cryptanalytically extracting bias-free Gated Linear Unit (GLU) feed-forward blocks from language models. This technique, which uses finite-difference curvature and paired obse…
-
llama.cpp releases include server improvements and performance optimizations · 8 sources tracked
The llama.cpp project has released several updates, including version b10331 which improves server functionality by correctly reporting the isolate working directory. Other recent releases, such as b10330 and earlier, h…
-
New lightweight detection transformer enables integer-only inference
Researchers have developed I-LW-DETR, a novel lightweight detection transformer that enables fully integer-only inference. This is a significant advancement for deploying such models on NPUs and microcontrollers, which …
-
FFNet introduces efficient convolutional mixer for vision tasks
Researchers have introduced FFNet, a novel convolutional mixer architecture designed for enhanced efficiency in computer vision tasks. FFNet reinterprets the Feed-Forward Network (FFN) component of Transformers as a mem…
-
Tessera system unlocks heterogeneous GPUs for AI workloads
A new system called Tessera has been developed to improve the performance and cost-efficiency of running large AI models on heterogeneous GPU clusters. Unlike previous methods that operated at a coarse granularity, Tess…
-
New method trains transformers for enhanced legibility and editability
Researchers have developed a method to train more legible transformer models by incorporating a per-channel variance floor as a loss metric. This approach encourages the model to use crisp, contextual detectors rather t…
-
New analysis unifies gradient descent convergence for deep neural networks
Researchers have developed a unified convergence analysis for various gradient descent optimization methods used in training deep neural networks. This new analysis applies to a broad range of optimizers, including Adam…
-
New structural interpretation of GELU and other activation functions proposed
Researchers have proposed a new structural interpretation of activation functions like GELU, ReLU, SiLU/Swish, and hard swish. This work views GELU not just as a stochastic gate output, but through a Gaussian complement…
-
New NC-FFN architecture enhances transformer interpretability and efficiency
Researchers have developed a novel parameter-neutral replacement for transformer feed-forward networks, termed NC-FFN, which utilizes explicit fuzzy set operations. This new architecture demonstrates strong parameter ef…
-
DREG regularization method shows superior accuracy in deep learning
Researchers have introduced DREG, a layer-wise Jacobian regularization technique that functions as a general-purpose penalty for neural networks. In a large-scale empirical study, DREG demonstrated superior accuracy com…
-
New research probes Transformer energy use, learned linearity, and training dynamics
Recent research explores the intricacies of Transformer models, focusing on their energy consumption, internal linear properties, and training dynamics. One paper introduces a scaling model to predict energy usage durin…
-
New IGLU activation function offers improved gradient flow
Researchers have introduced IGLU, a novel parametric activation function for deep neural networks designed to improve gradient flow and optimization stability. Derived from a mixture of GELU gates under a half-normal di…
-
New neural network architectures tackle complex scientific computing problems · 8 sources tracked
Researchers are developing novel neural network architectures to solve complex partial differential equations (PDEs) and model dynamical systems. These include structure-oriented randomized neural networks (SO-RaNN) for…
-
Adam vs. SGD: No single factor explains performance gap, study finds
A new research paper explores the performance gap between the Adam and SGD optimization algorithms, finding that no single factor consistently explains the difference. The study indicates that the gap arises from comple…
-
Apple M4 Max GPU's Tensor Compute Path Emulated, Not Accelerated
Researchers have reverse-engineered the Metal 4.1 tensor compute path on Apple's M4 Max GPU, revealing that the fp8 matmul2d operation is emulated rather than hardware-accelerated. This means the operation runs on the G…
-
New algorithm offers robust learning for nonlinear AI models
Researchers have developed a novel algorithm for robustly learning Gaussian Single Index Models (SIMs) even when faced with heavy-tailed noise and adversarial corruption. This new method provides the first robust recove…
-
Paper analyzes floating-point neural network expressivity
Researchers have published a paper exploring the expressive power of neural networks operating with floating-point arithmetic, moving beyond theoretical models that assume exact real numbers. The study introduces a fram…
-
New MoA FFN Design Enhances LLM Expressivity and Scaling
Researchers have introduced a novel feedforward network (FFN) design called Mixture of Activations (MoA) for large language models (LLMs). MoA utilizes token-adaptive activation mixing, allowing different activation fun…
-
Activation functions enable neural networks to model complex, non-linear patterns
Neural networks rely on activation functions to introduce non-linearity, enabling them to model complex patterns beyond simple linear relationships. Without these functions, even deep networks would collapse into equiva…