decoder-only language-model FFNs

SwiGLU's open positive tail not necessary for language models, study finds

Researchers have investigated the necessity of SwiGLU's open positive tail in decoder-only language models. They introduced MemGLU as a closed-tail alternative and found that across multiple pretraining runs, MemGLU per…