Mixtral

LLM pre-training research explores sparse vs. dense and low-rank methods

Two new research papers explore efficient pre-training methods for large language models. The first paper compares dense and sparse Mixture-of-Experts (MoE) transformer architectures at a small scale, finding that MoE m…