A new arXiv paper evaluates quantized versions of the Qwen2.5 Coder large language model to determine the optimal balance between file size, precision, and energy consumption per solved task on a MacBook Air M2. The study found that for smaller file sizes (1-2 GB), the q4_K_M quantization of the 1.5B and 3B models offered the best performance in terms of correctness and energy efficiency. Larger models at lower precision, like the 7B-q2_K model, did not provide significant gains in solved tasks despite requiring substantially more energy. AI
IMPACT Provides guidance for developers on selecting efficient quantized code LLMs for local deployment, balancing performance with resource constraints.
RANK_REASON The cluster contains an academic paper detailing research findings on LLM performance. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →