vLLM integrates Tenstorrent hardware for LLM serving via new TT Plugin

vLLM has released a new TT Plugin that enables the use of Tenstorrent hardware for serving large language models. This plugin integrates Tenstorrent's accelerators into the vLLM framework via a standard platform plugin mechanism. The integration allows for the serving of various models, including text-only and multimodal ones, using an OpenAI-compatible API without altering existing client code. The plugin leverages Tenstorrent's unique mesh architecture, expressing differences like a phase-constrained scheduler and custom data-parallel topologies directly within the compiled mesh program. AI

IMPACT Enables broader hardware options for LLM deployment, potentially improving cost-efficiency and performance.

RANK_REASON The item describes a software plugin that enables existing LLM serving infrastructure (vLLM) to utilize specific hardware (Tenstorrent), which falls under the 'tool' category.

Read on Mastodon — mastodon.social →

• Galaxy
• Gemma 3
• Llama 3.2 Vision
• Mistral 3
• OpenAI
• QuietBox
• Qwen-3.6
• Qwen3.6-27B
• Qwen VL
• Tenstorrent
• TT-Metal
• vLLM
• vLLM TT Plugin

• infra
• product

AI-generated summary · Google Gemini · from 2 sources. How we write summaries →