Small local LLMs struggle with token limits, not understanding, in Japanese classification tasks

A recent evaluation of small, locally run language models revealed that their performance on Japanese classification tasks was significantly hampered not by a lack of understanding, but by token budget limitations. Models like LFM2.5-2.6B and Phi 4 Mini often failed to produce complete answers within a 512-token limit, leading to them being scored as incorrect despite having accurate reasoning. When token budgets were increased, these models showed improved answer rates and accuracy, though some still hit caps or produced unparseable output. Ultimately, even with these improvements, the tested local models did not surpass the accuracy of a hosted API on these specific tasks, with the API's advantage stemming from its inability to format-fail. AI

IMPACT Highlights the critical role of token budgeting and output formatting in the practical application of smaller, locally run LLMs.

RANK_REASON The item details an evaluation of multiple small language models on specific tasks, comparing their performance and identifying limitations. [lever_c_demoted from research: ic=1 ai=1.0]

Read on dev.to — LLM tag →

• Gemma 4-E2B
• Jevíčko
• LFM2.5-2.6B
• llama.cpp
• MiniCPM5 2B
• Phi 4 Mini
• SemIf 4B

• model release
• product

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →