TheBloke

Rust engine streams Mixtral 8x7B on cheap VMs

A new Rust-based inference engine called MER allows for efficient streaming of large language models like Mixtral 8x7B from NVMe storage onto less powerful and cheaper virtual machines. This approach bypasses the need f…