Small Language Models Fail Microtask Eligibility Benchmarks

A new arXiv paper investigates the eligibility of small language models (SLMs) for microtasks within agent harnesses. The study found that even with optimal configurations and FP16 precision, none of the tested Qwen models met the defined eligibility thresholds. Quantization to 4-bit precision further degraded performance, with the gap tracking model size rather than precision. The research suggests that SLMs should be used behind a baseline system that meets the required thresholds, rather than as a primary component for these microtasks. AI

IMPACT Highlights limitations of current SLMs for agent harnesses, suggesting a need for better baselines or model improvements.

RANK_REASON Academic paper published on arXiv detailing model performance on specific benchmarks. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.AI →

• arXiv
• BM25
• Llama 3
• Qwen

• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →