Brief

A user on Reddit shared performance benchmarks for the Qwen 3.6 large language model, specifically testing the 27B and 35B parameter versions. The tests were conducted using a setup with two RTX PRO 6000 GPUs and the latest stable VLLM backend. Results indicate varying throughputs depending on concurrency levels and whether multi-turn prompting (MTP) was enabled, with the 35B model achieving up to 3500 tokens per second at 128 concurrency. AI

Amazon SageMaker AI now supports bidirectional streaming, enabling real-time, two-way communication between clients and model containers. This feature, combined with vLLM's Realtime API, allows for continuous audio streaming and simultaneous transcription. The integration is demonstrated by deploying Mistral AI's Voxtral-Mini-4B-Realtime-2602 model for efficient speech-to-text applications. AI

Running large open-source language models locally can lead to out-of-memory errors, even if the model's weights seem to fit within the available VRAM. This is primarily due to the significant memory required for the KV cache, which scales with context length, and intermediate activation memory during inference. Developers can address these issues by profiling memory usage with tools like PyTorch's memory snapshot, applying appropriate quantization techniques to model weights and the KV cache, and managing memory fragmentation. AI

A blog post details the transition of vLLM from version 0 to version 1, focusing on its accuracy before reinforcement learning corrections. The post highlights the model's performance and improvements in this area. AI

This article provides a guide on deploying the Mistral 7B language model on a GPU server using the vLLM framework. It is aimed at users with limited budgets and resources who need to set up a self-hosted LLM solution. The recommended setup involves Mistral-7B-Instruct-v0.3 and a virtual machine, detailing the inference process on cloud servers with NVIDIA RTX GPUs. AI

This article provides a guide for optimizing vLLM deployments, focusing on three critical configuration decisions that impact performance and cost. It details how static KV cache allocation can lead to GPU out-of-memory errors and emphasizes the importance of selecting the right serving framework, managing memory budgets for KV cache versus model weights, and configuring batching strategies like chunked prefill and prefix caching. The guide also outlines common failure modes and offers architectural insights for effective vLLM operation. AI

This article details how to achieve end-to-end observability for large language model inference servers like vLLM and TGI. It highlights that standard observability tools fall short due to unique LLM serving characteristics such as variable latency, dynamic batching, and the critical role of the KV cache. The author proposes a layered approach, correlating user-facing token rendering with underlying GPU silicon metrics, and provides specific signals to monitor at each layer, from business costs down to GPU hardware. AI

Unsloth has released version v0.1.405-beta, introducing significant performance enhancements and new features. The update includes up to 2x faster GGUF inference through MTP speculative decoding and adds API calling support for services like OpenAI and Anthropic, enabling features such as web search and code execution. Additionally, Unsloth now offers experimental MLX inference for Mac users and improved support for non-English languages, alongside various security and UI/UX improvements. AI

A Hugging Face model repository, froggeric/Qwen-Fixed-Chat-Templates, has been updated with significant improvements to its chat templates for Qwen 3.5 and 3.6 models. These updates address issues such as "empty think" poisoning, system prompt logic traps, and KV cache inconsistencies. The changes aim to enhance the model's ability to use tools, transition between thinking and conversational responses, and maintain a consistent memory during multi-step processes. AI

The Jiunsong/supergemma4-26b-uncensored-gguf-v2 model is now available for use with various popular AI libraries and applications. These include llama-cpp-python, llama.cpp, vLLM, Ollama, Unsloth Studio, and Pi. Detailed instructions and code snippets are provided for integrating the model into local applications and servers, enabling users to run inference directly or via OpenAI-compatible APIs. AI

Anyscale has introduced a new fault tolerance feature for its vLLM serving engine, integrated with Ray Serve. This enhancement specifically addresses the challenges of deploying large Mixture-of-Experts (MoE) models, which are sharded across multiple GPUs. The new system can now identify and restart entire groups of GPUs that form a data-parallel (DP) group when a single GPU within that group fails, preventing the entire deployment from becoming unavailable. AI

Researchers at Together AI have developed AutoJudge, a novel method to accelerate large language model inference. This technique automates the curation of task-specific datasets, enabling lossy speculative decoding without manual annotation. AutoJudge identifies critical tokens that impact downstream quality, achieving up to a 2x speedup over standard speculative decoding with minimal accuracy loss. AI

Together AI is enhancing its cloud platform to support advanced reinforcement learning (RL) pipelines, integrating TorchForge and Monarch for distributed training. The platform now offers low-latency GPU communication and heterogeneous scheduling for mixed CPU/GPU workloads, crucial for complex RL tasks. New integrations with Together CodeSandbox and Code Interpreter allow RL agents to interact with tools and execute code, expanding their capabilities beyond traditional game-playing scenarios. AI

A developer has created an open-source AI-powered web service that integrates FastAPI for APIs, Pydantic-AI for agent construction, and Model Context Protocol (MCP) servers for tools. The service allows users to query information from sources like Hacker News and web search, presenting ranked trend cards with summaries. It supports various local LLM configurations and is containerized with Docker for production deployment. AI