A common issue with vLLM's tool-calling functionality, particularly when using models like Kimi K2.6 or DeepSeek, stems from a default server configuration that leads to a significant rate of invalid JSON outputs. Ubicloud's testing revealed that vLLM, by default, only achieved a 92% accuracy for schema-valid tool calls, which severely impacts multi-step agent processes. The problem is exacerbated by the lack of a "strict" parameter in many tool definitions, preventing vLLM from enforcing schema adherence. A fix is available in vLLM versions 0.31 and later through the `--tool-strict-level` server flag, which ensures that tool calls are properly constrained and validated. AI
IMPACT Fixing vLLM's default tool-calling configuration is crucial for reliable agent performance and reduces model-specific blame.
RANK_REASON The item discusses a specific technical issue and its solution within the vLLM framework, impacting tool-calling functionality.
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
