InferPrice

llama.cpp MoE cache nearly doubles decode speed for large models

The llama.cpp project has introduced a new Mixture of Experts (MoE) cache feature that significantly boosts decoding speed for large models that do not fit entirely into GPU memory. This cache stores frequently used exp…