Strata enables large MoE models on consumer hardware using efficient memory management

Strata, a fork of llama.cpp, enables running large Mixture of Experts (MoE) models on consumer hardware by intelligently managing memory. It keeps only the most frequently used "hot" experts in VRAM, while parking the rest in cheaper system DDR memory. This approach allows organizations to deploy sophisticated AI models using machines with more RAM than VRAM, reducing hardware costs and keeping sensitive data on-premises. AI

IMPACT Enables running larger AI models on less expensive hardware, potentially lowering the barrier to entry for local AI deployments.

RANK_REASON The item describes a software tool (Strata) that optimizes existing hardware for running AI models, rather than a new model release or fundamental research.

Read on dev.to — LLM tag →

• Avalon
• central processing unit
• CUDA
• GGUF
• llama.cpp
• Metal
• Mixture of Experts
• NVM Express
• RTX 5080
• Strata
• VRAM
• Vulkan

• infra
• product

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →