Mixture of Depths
PulseAugur coverage of Mixture of Depths — every cluster mentioning Mixture of Depths across labs, papers, and developer communities, ranked by signal.
3 day(s) with sentiment data
-
Mixture-of-Depths dynamically allocates transformer compute, saving ~50% FLOPs
Researchers have developed a novel technique called Mixture-of-Depths (MoD) that dynamically allocates computational resources within transformer models. By introducing a small router in each block, MoD identifies and p…
-
MixCompress framework introduces Mixture-of-Experts for efficient image compression
Researchers have introduced MixCompress, a novel framework for learned image compression that addresses the limitations of storing separate models for each compression rate. This new approach utilizes a sparse Mixture-o…
-
New research explores unified routing for adaptive LLM efficiency · 2 sources tracked
Two new research papers explore methods to optimize the efficiency of large language models by dynamically adjusting computational resources based on token complexity. The first paper, "Linear Attention Architectures," …
-
Open-source framework accelerates LLM training with MoE/MoD
A developer has created an open-source PyTorch framework designed for training large language models with Mixture of Experts (MoE) and Mixture of Depths (MoD) architectures. The framework incorporates custom CUDA kernel…