Local LLM optimization: Step-3.7-Flash gains 2.4x speed, MTP breaks vision

A developer has optimized the Step-3.7-Flash (198B-A11B vision MoE) model for local hardware, achieving significant performance gains. By ensuring the model's largest quantization (IQ3_XXS) fits entirely within the 96GB VRAM across four 3090 GPUs, they observed a 2.4x speed increase compared to a higher quantization (IQ4_XS) that spilled data to the CPU. Additionally, the developer found that the model's speculative decoding feature (MTP) is incompatible with its vision capabilities, causing hard aborts when processing image tokens. AI

IMPACT Demonstrates how VRAM capacity significantly impacts local LLM performance, influencing hardware choices and model quantization strategies.

RANK_REASON Developer's optimization of an existing open-source model for local hardware.

Read on r/LocalLLaMA →

• 198B-A11B vision MoE
• 4×3090
• IQ3_XXS
• llama.cpp
• NVIDIA
• Step-3.7-Flash

• infra
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →