SwiGLU
PulseAugur coverage of SwiGLU — every cluster mentioning SwiGLU across labs, papers, and developer communities, ranked by signal.
9 day(s) with sentiment data
-
New research diagnoses rank collapse in decoder-only transformers
A new research paper published on arXiv details a mechanistic diagnostic for understanding rank collapse in post-norm decoder transformers. The study analyzes how causal attention in these models leads to high-similarit…
-
SwiGLU's open positive tail not necessary for language models, study finds
Researchers have investigated the necessity of SwiGLU's open positive tail in decoder-only language models. They introduced MemGLU as a closed-tail alternative and found that across multiple pretraining runs, MemGLU per…
-
UniMoMo framework compresses MoE recommendation models for faster inference
Researchers have developed UniMoMo, a post-training compression framework designed to accelerate large recommendation models that utilize mixture-of-experts (MoE) layers. This method groups experts based on their functi…
-
Lego Analogy Deciphers Modern GPT Architectures and Efficiency Gains
This article uses a Lego analogy to explain the inner workings of modern GPT architectures, detailing how individual tokens are processed from input to output. It breaks down key refinements like RoPE, RMSNorm, and SwiG…
-
llama.cpp releases include server improvements and performance optimizations · 8 sources tracked
The llama.cpp project has released several updates, including version b10331 which improves server functionality by correctly reporting the isolate working directory. Other recent releases, such as b10330 and earlier, h…
-
New Prox method boosts LLM efficiency via FFN activation sparsity
Researchers have developed a new training-free method called Prox to improve the efficiency of large language models (LLMs) by sparsifying their feed-forward networks (FFNs). Prox utilizes the intermediate state of SwiG…
-
Kimi K3 unveils architectural innovations for long-context and agent tasks
Kimi K3 has released its technical report detailing significant architectural innovations aimed at improving the efficiency and scalability of large language models, particularly for long-context tasks and agentic opera…
-
New research probes truth representation in small language models
Researchers have explored the internal mechanisms of truth representation in small language models, building upon prior work that identified universal truth subspaces. Their investigation reveals that the dimensionality…
-
New framework enhances machine analysis of human emotions from facial expressions
Researchers have developed a novel framework for Affective Behaviour Analysis, focusing on improving the accuracy of machines in inferring human emotional states from facial expressions. The proposed method introduces c…
-
Amplitude Gating improves LLM structured output without retraining
Researchers have developed a new method called Amplitude Gating (AG) to improve the structured output of large language models during inference without retraining. This technique modulates activation magnitudes within f…
-
New kernels boost LLM inference speed by fusing SwiGLU activations
Researchers have developed new techniques to accelerate the inference of large language models (LLMs) by fusing SwiGLU activation functions directly into GEMM operations at the tile level. These methods, implemented usi…
-
Modern LLM Transformer Blocks Evolve with RMSNorm, GQA, and MoE
Modern Transformer blocks in Large Language Models (LLMs) have evolved beyond the original 2017 design to improve training stability, context length, inference efficiency, and model capacity. Key advancements include th…
-
New QG-MIL architecture enhances medical imaging analysis accuracy
Researchers have developed QG-MIL, a novel gated transformer aggregator designed to improve the stability and accuracy of multiple instance learning (MIL) in medical imaging. This new architecture addresses issues of ov…
-
xFormers library enables memory-efficient Transformer models on GPUs
This tutorial demonstrates how to build memory-efficient Transformer models using the xFormers library on GPUs. It covers implementing and comparing memory-efficient attention with standard attention, analyzing techniqu…
-
New neural network architectures tackle complex scientific computing problems · 8 sources tracked
Researchers are developing novel neural network architectures to solve complex partial differential equations (PDEs) and model dynamical systems. These include structure-oriented randomized neural networks (SO-RaNN) for…
-
AI research requires discipline, foundational knowledge, and a beginner's mindset
Becoming a successful AI researcher requires a blend of consistent effort and hands-on building, akin to a meditative practice where dedication is key even without immediate insights. Focusing on fundamental concepts ra…
-
New MoA FFN Design Enhances LLM Expressivity and Scaling
Researchers have introduced a novel feedforward network (FFN) design called Mixture of Activations (MoA) for large language models (LLMs). MoA utilizes token-adaptive activation mixing, allowing different activation fun…
-
Transformer LLM Architectures Converge on Standard Stack
A recent analysis of 53 large language models from 2017 to 2025 reveals a significant convergence in transformer architectures. Key elements of this de facto standard include pre-normalization (RMSNorm), Rotary Position…
-
IBM releases Granite 4.1 LLMs with 512K context and Apache 2.0 license
IBM has released the Granite 4.1 family of large language models, comprising 3B, 8B, and 30B parameter versions. These models were trained on approximately 15 trillion tokens through a five-stage pre-training process th…
-
MLP skip connections can't be absorbed into residual-free models
Researchers have investigated whether a skip connection around a single-hidden-layer MLP can be absorbed into a residual-free MLP of the same width. They found that for certain activation functions like ReLU^2 and ReGLU…