Learnable activations (LA)

New MoA FFN Design Enhances LLM Expressivity and Scaling

Researchers have introduced a novel feedforward network (FFN) design called Mixture of Activations (MoA) for large language models (LLMs). MoA utilizes token-adaptive activation mixing, allowing different activation fun…