vLLM
PulseAugur coverage of vLLM — every cluster mentioning vLLM across labs, papers, and developer communities, ranked by signal.
- used by graphics processing unit 90%
- used by H.1000 Gnome 80%
- used by llama-cpp-python 70%
- used by Fp8 70%
- used by Horizon 2020 70%
- uses Anyscale, Inc. 70%
- competes with Text Generation Inference 60%
- used by Mlx 60%
- uses LM Studio 60%
- affiliated with Anyscale, Inc. 50%
- affiliated with LM Studio 50%
- affiliated with llama-cpp-python 50%
- 2026-05-15 product_launch vLLM released version 0.21.1rc0.
15 天有情绪数据
-
DeepSeek V4 发布，拥有 1.6T MoE、1M 上下文和更低成本
DeepSeek V4 是一个开放权重模型系列，已发布，采用 1.6 万亿参数的专家混合（MoE）架构，每个 token 只激活 490 亿参数。该新模型拥有 100 万 token 的上下文窗口，并显著降低了推理成本，由于混合注意力（Hybrid Attention）等创新，成本比前代产品降低高达 73%。V4 系列可在 Hugging Face 上获取，其质量可与 GPT-5.4 和 Claude Opus 4.6 等领先模型相媲…
-
PyTorch tutorial simplifies distributed AI model inference
This article explains distributed inference techniques for large AI models using PyTorch. It details how to implement Data Parallelism (DP), Tensor Parallelism (TP), and Pipeline Parallelism (PP) with minimal code. The …
-
vLLM CPU backend setup detailed by new contributor
A new contributor to vLLM has documented challenges and solutions for setting up the project's CPU backend. The process requires specific GCC versions and hidden build dependencies like setuptools_scm, which are not cle…
-
PreFT method boosts LLM serving throughput with prefill-only finetuning
Researchers have developed PreFT, a novel parameter-efficient finetuning method designed to improve the efficiency of serving personalized large language models. PreFT optimizes for serving throughput by applying adapte…
-
Docker Model Runner simplifies local AI development with integrated LLM support
Docker has integrated a new feature called Model Runner directly into Docker Desktop, simplifying local AI development. This tool allows users to pull and run various language models, such as Llama 3.1 and Phi-3-mini, u…
-
KVServe framework slashes LLM serving latency with adaptive compression
Researchers have developed KVServe, a novel framework designed to optimize communication efficiency in disaggregated LLM serving systems. KVServe addresses the bottleneck caused by KV cache data crossing network and sto…
-
New framework optimizes LLM inference energy use on multi-GPU systems
Researchers have developed EnergyLens, a framework designed to optimize the energy consumption of large language models (LLMs) during inference on multi-GPU systems. This tool addresses the challenge of predicting and r…
-
AMD invests $3.6M in AI dev clusters to boost ROCm ecosystem
AMD is making significant efforts to support the open-source AI community, particularly with its ROCm software stack. The company has recently provided access to interconnected MI355X development clusters, valued at $3.…
-
WSL2 vllm fails Qwen2.5-7B-1M on 6GB VRAM, Windows transformers succeed
A developer encountered unexpected memory limitations when attempting to run the Qwen2.5-7B-1M model on a consumer laptop with 6GB of VRAM. While the Windows "transformers" library could handle a 4k context by spilling …
-
Local AI tools boost LLM speeds with new prediction and decoding techniques
Recent updates in the local AI community are enhancing inference speeds and providing practical benchmarks for open-weight models. The llama.cpp project now supports Multi-Token Prediction (MTP), which has shown a 40% s…
-
Superhuman and Databricks build 200K QPS AI inference platform
Superhuman and Databricks engineers collaborated to build a high-throughput inference platform capable of handling over 200,000 queries per second. This joint effort modernized Superhuman's serving stack, migrating from…
-
Self-hosted LLM with Nextcloud, LocalAI, and vLLM sees response time optimizations
A self-hosted Nextcloud instance was optimized for faster LLM response times by implementing LocalAI and vLLM. The team identified unpredictable latency issues and developed solutions to improve performance. This setup …
-
Gemma-4-31B model hits 463K tokens/sec on TPU v6e-4 benchmarks
A performance report details the Gemma-4-31B model's capabilities on Cloud TPU v6e-4 hardware, achieving a peak prefill throughput of 463,345 tokens/sec. The benchmarks indicate that the dense 31B model offers comparabl…
-
Local AI models lag hosted APIs due to complex setup and lack of polish
Armin Ronacher argues that while significant progress has been made in running AI models locally, the user experience for developers, particularly with coding agents, remains frustratingly complex. He highlights the gap…
-
New research explores speculative decoding for faster LLM inference
Multiple research papers published on arXiv explore advancements in speculative decoding for Large Language Models (LLMs). These studies focus on improving inference speed and efficiency by using a smaller "draft" model…
-
Visual Para-Thinker introduces parallel reasoning to multimodal LLMs
Researchers have introduced Visual Para-Thinker, a novel framework for parallel reasoning in multimodal large language models (MLLMs). This approach shifts from vertical scaling of reasoning depth to a parallel strategy…
-
vLLM project optimizes DeepSeekv4 performance, merging model support PR
The vLLM project maintainers have rapidly integrated support for the new DeepSeekv4 model, merging their initial pull request over the weekend. This swift action highlights the project's focus on optimizing performance …
-
vLLM releases v0.20.2 with automated Docker Hub image publishing
The vLLM project has released version 0.20.2, which includes an automated process for publishing Docker Hub release images. This update aims to streamline the deployment and accessibility of vLLM's inference engine.
-
Seven small coding AI models offer local development power in 2026
The article highlights seven small coding AI models suitable for local development, emphasizing their efficiency and privacy benefits. These models, including OpenAI's gpt-oss-20b and Microsoft's Phi-3.5-mini-instruct, …
-
vLLM V1引擎重写在后端修复后实现与V0的对等
Hugging Face的vLLM团队详细介绍了如何将他们新的V1引擎与V0参考模型对齐的过程，重点在于确保后端对等，然后再处理强化学习（RL）目标的变化。他们识别并修复了四个关键问题：处理已处理的logprobs的方式、V1特有的运行时默认值、inflight权重更新路径以及使用fp32作为最终投影层。这些修正对于恢复后端行为以匹配V0参考模型至关重要，从而能够准确评估RL目标调整。