Brief

This article discusses how to identify and address GPU waste within Kubernetes clusters, a problem that often goes unnoticed due to seemingly healthy utilization metrics. It highlights that inefficient GPU usage can occur even when overall cluster utilization appears normal. The piece aims to provide methods for detecting these hidden inefficiencies. AI

Modal has developed a system to achieve truly serverless GPUs for AI inference, addressing the challenge of rapidly scaling resources to meet variable demand. Their approach involves maintaining cloud buffers of idle GPUs, a custom filesystem for lazy container image serving, and efficient checkpoint/restore mechanisms for both CPU and GPU processes. This engineering effort, developed over five years, reduces AI inference replica scaling time from tens of minutes to mere seconds, aiming to maximize GPU Allocation Utilization. AI

Modal has introduced an autoscaling feature for GPUs designed to support AI research agents. This new capability allows agents to dynamically provision and release compute resources as needed, addressing the challenge of managing unpredictable research workloads and associated costs. The system demonstrated its effectiveness by completing OpenAI's Parameter Golf challenge significantly faster than a traditional workstation, highlighting its efficiency and cost-effectiveness for AI-driven research. AI

A new method called MTP (Multi-Token Prediction) has been developed to accelerate token generation in AI models. This technique involves predicting multiple future tokens simultaneously and then having the main model verify them in parallel. However, MTP requires a significant increase in VRAM, which can lead to slower generation or reduced context size on GPUs with limited memory. The technique does not appear to reduce model hallucinations. AI

This article provides a guide for optimizing vLLM deployments, focusing on three critical configuration decisions that impact performance and cost. It details how static KV cache allocation can lead to GPU out-of-memory errors and emphasizes the importance of selecting the right serving framework, managing memory budgets for KV cache versus model weights, and configuring batching strategies like chunked prefill and prefix caching. The guide also outlines common failure modes and offers architectural insights for effective vLLM operation. AI

Manus has launched Scheduled Tasks 2.0, transforming basic reminders into intelligent agents capable of maintaining context and autonomously updating web applications. Meanwhile, the United Arab Emirates aims to generate 40% of Dubai's GDP from AI by 2031 through widespread automation of legislation and public services. In Poland, only 9% of companies possess the necessary GPU infrastructure for advanced AI, leading to a common bottleneck in pilot project implementation. AI

Researchers have developed WarmServe, a new system designed to improve the efficiency of serving multiple large language models (LLMs) on shared GPU clusters. WarmServe utilizes a one-for-many GPU prewarming strategy, proactively loading model parameters based on predicted workload patterns. This approach aims to reduce the time-to-first-token (TTFT) degradation often seen in multi-LLM serving systems. Evaluations indicate WarmServe can significantly decrease tail TTFT and increase request throughput compared to existing methods. AI

Researchers have developed FlashSinkhorn, a new GPU-accelerated solver for entropic optimal transport (EOT) that significantly reduces memory input/output operations. By rewriting stabilized log-domain Sinkhorn updates to mimic the normalization process in transformer attention, FlashSinkhorn enables fused kernels that stream data through on-chip SRAM. This approach achieves substantial speedups, up to 32x for forward passes and 161x end-to-end, compared to existing methods on A100 GPUs for tasks like point-cloud OT. AI

Researchers have developed GEM, a framework designed to optimize the mapping of experts to GPUs in Mixture-of-Expert (MoE) AI models. This new approach accounts for variability in GPU performance, aiming to reduce inference latency by strategically placing experts. GEM's strategy involves distributing experts to ensure GPUs finish processing layers concurrently, thereby mitigating slowdowns caused by slower GPUs or overloaded experts. Experiments indicate that GEM can improve end-to-end latency by an average of 7.9%, with some cases showing improvements up to 16.5%. AI

Anyscale has introduced a new fault tolerance feature for its vLLM serving engine, integrated with Ray Serve. This enhancement specifically addresses the challenges of deploying large Mixture-of-Experts (MoE) models, which are sharded across multiple GPUs. The new system can now identify and restart entire groups of GPUs that form a data-parallel (DP) group when a single GPU within that group fails, preventing the entire deployment from becoming unavailable. AI

The Together AI kernels team, including researchers Dan Fu and Tri Dao, developed FlashAttention, a software layer that significantly optimizes GPU performance for AI models. This breakthrough, achieved by applying database system principles to GPU memory movement, resulted in 2-3x speedups, challenging the notion that transformer attention was already fully optimized. The team's subsequent work, including the ThunderKittens library, aims to accelerate kernel development for new hardware like NVIDIA's Blackwell GPUs, addressing the critical software-hardware gap in AI infrastructure. AI

Training large foundation models necessitates distributing the workload across numerous GPUs housed in multiple interconnected machines, a process known as multi-node training. This approach is essential for handling models with billions or trillions of parameters that exceed the memory capacity of single servers and would otherwise take months to train. Effective multi-node training relies on sophisticated parallelism strategies, high-speed network interconnects, and robust fault tolerance mechanisms to ensure efficient computation and progress. AI