Gelu
PulseAugur coverage of Gelu — every cluster mentioning Gelu across labs, papers, and developer communities, ranked by signal.
- instance of SwiGLU 90%
- instance of Swish 70%
- used by LayerNorm 70%
- used by Softmax 70%
- used by multilayer perceptron 70%
- instance of sigmoid function 70%
- instance of hyperbolic tangent 70%
- competes with Swish 70%
- used by Swish 70%
- competes with Silu Activation Function 60%
- affiliated with Swish 50%
- used by SwiGLU 50%
4 day(s) with sentiment data
-
Research paper shows Transformers can compile into generic deep neural networks
A new research paper demonstrates that Transformer models can be compiled into generic deep neural networks, challenging their classification as a special model class. The study shows that Transformer blocks, including …
-
Activation bias in neural networks impacts learning dynamics, research finds
A new research paper published on arXiv explores the concept of "activation bias" in neural networks, a phenomenon where common activation functions like ReLU, GELU, Swish, and Sigmoid predominantly produce positive out…
-
New ZonoGPT method verifies large transformer models like GPT-2 Medium
Researchers have developed ZonoGPT, a new abstract domain designed for verifying large transformer-based models. This approach maintains a space complexity independent of network depth and uses block-specific fused tran…
-
New methods enhance LLM efficiency via advanced quantization techniques · 4 sources tracked
Researchers are developing new methods to improve the efficiency of large language models through quantization-aware training and post-training quantization. Q-PACE, a new approach, dynamically allocates precision to mo…
-
New SG-Blend activation function improves neural network robustness
Researchers have introduced SG-Blend, a novel adaptive activation function designed to improve neural network representations. SG-Blend interpolates between Swish and GELU activation functions, allowing each layer to le…
-
New method uses FFN curvature to recover hidden transformer structures
Researchers have developed a novel method for cryptanalyzing transformer feed-forward networks (FFNs) by examining their curvature. This technique exploits a second-order leakage channel, specifically projected input He…
-
New method extracts bias-free GLU blocks from language models
Researchers have developed a new method for cryptanalytically extracting bias-free Gated Linear Unit (GLU) feed-forward blocks from language models. This technique, which uses finite-difference curvature and paired obse…
-
llama.cpp releases include server improvements and performance optimizations · 8 sources tracked
The llama.cpp project has released several updates, including version b10331 which improves server functionality by correctly reporting the isolate working directory. Other recent releases, such as b10330 and earlier, h…
-
New lightweight detection transformer enables integer-only inference
Researchers have developed I-LW-DETR, a novel lightweight detection transformer that enables fully integer-only inference. This is a significant advancement for deploying such models on NPUs and microcontrollers, which …
-
FFNet introduces efficient convolutional mixer for vision tasks
Researchers have introduced FFNet, a novel convolutional mixer architecture designed for enhanced efficiency in computer vision tasks. FFNet reinterprets the Feed-Forward Network (FFN) component of Transformers as a mem…
-
Tessera system unlocks heterogeneous GPUs for AI workloads
A new system called Tessera has been developed to improve the performance and cost-efficiency of running large AI models on heterogeneous GPU clusters. Unlike previous methods that operated at a coarse granularity, Tess…
-
New method trains transformers for enhanced legibility and editability
Researchers have developed a method to train more legible transformer models by incorporating a per-channel variance floor as a loss metric. This approach encourages the model to use crisp, contextual detectors rather t…
-
New analysis unifies gradient descent convergence for deep neural networks
Researchers have developed a unified convergence analysis for various gradient descent optimization methods used in training deep neural networks. This new analysis applies to a broad range of optimizers, including Adam…
-
New structural interpretation of GELU and other activation functions proposed
Researchers have proposed a new structural interpretation of activation functions like GELU, ReLU, SiLU/Swish, and hard swish. This work views GELU not just as a stochastic gate output, but through a Gaussian complement…
-
New NC-FFN architecture enhances transformer interpretability and efficiency
Researchers have developed a novel parameter-neutral replacement for transformer feed-forward networks, termed NC-FFN, which utilizes explicit fuzzy set operations. This new architecture demonstrates strong parameter ef…
-
DREG regularization method shows superior accuracy in deep learning
Researchers have introduced DREG, a layer-wise Jacobian regularization technique that functions as a general-purpose penalty for neural networks. In a large-scale empirical study, DREG demonstrated superior accuracy com…
-
New research probes Transformer energy use, learned linearity, and training dynamics
Recent research explores the intricacies of Transformer models, focusing on their energy consumption, internal linear properties, and training dynamics. One paper introduces a scaling model to predict energy usage durin…
-
New IGLU activation function offers improved gradient flow
Researchers have introduced IGLU, a novel parametric activation function for deep neural networks designed to improve gradient flow and optimization stability. Derived from a mixture of GELU gates under a half-normal di…
-
New neural network architectures tackle complex scientific computing problems · 8 sources tracked
Researchers are developing novel neural network architectures to solve complex partial differential equations (PDEs) and model dynamical systems. These include structure-oriented randomized neural networks (SO-RaNN) for…
-
Adam vs. SGD: No single factor explains performance gap, study finds
A new research paper explores the performance gap between the Adam and SGD optimization algorithms, finding that no single factor consistently explains the difference. The study indicates that the gap arises from comple…