graphics processing unit

Recent developments

• 2026-10-19 — Nvidia mobilizes $500B+ for AI infrastructure financing
• 2026-10-19 — VIDRAFT trains LLM from scratch on NPUs, bypassing GPUs
• 2026-10-19 — vLLM introduces push-based KV transfer to cut LLM inference latency
• 2026-10-18 — New system optimizes MoE LLM inference with CPU-GPU collaboration
• 2026-10-16 — NVIDIA MIG enables GPU partitioning for multiple AI and container workloads
• 2026-10-02 — New SparLeak attack exploits sparse attention in LLMs to steal private data

Frequently asked

How are GPUs being optimized for large language model (LLM) inference this quarter?

This quarter, significant strides are being made in optimizing GPUs for LLM inference through software and architectural improvements. vLLM has introduced a push-based Key-Value cache transfer mechanism, improving time-to-first-token and time-per-output-token by up to 3x and 30% respectively. New systems are also optimizing Mixture-of-Experts (MoE) LLM inference by intelligently assigning experts to either CPU or GPU, dynamically minimizing latency. These efforts aim to maximize throughput and reduce latency for real-world AI applications.

What are the main infrastructure challenges facing GPU deployment for AI?

The primary infrastructure challenges include efficient GPU utilization, managing immense power demands, and high operational costs. Kubernetes GPU underutilization, often as low as 5%, is being addressed with solutions like MIG and time-slicing. The energy requirements for large AI clusters are so substantial that countries like China are prioritizing grid capacity and advanced cooling systems. The debate between renting vs. owning GPUs also highlights that on-premise private clusters can become more cost-effective than hyperscalers after about three years, due to significant cloud GPU expenses and debugging time.

Are Neural Processing Units (NPUs) becoming a competitive alternative to GPUs for AI training?

Yes, NPUs are emerging as a significant competitor. A startup named VIDRAFT has demonstrated the ability to pre-train large language models entirely on inference-only NPUs, bypassing traditional GPU reliance. Imagination Technologies has also unveiled E-series GPUs that integrate AI capabilities directly, featuring a 'Neural Core' for tasks like AI upscaling and LLM prefill. While GPUs remain central, these developments signal a growing diversification in AI hardware, with NPUs offering specialized advantages for certain AI workloads, particularly in power efficiency and on-device adaptation.

How is GPU technology being applied in robotics and scientific computing?

GPU technology is making significant advancements in robotics and scientific computing. Researchers have developed GPU-accelerated adaptive control frameworks for robots, allowing them to learn and reduce uncertainty in real-time. A GPU-resident pipeline optimizes multi-robot 3D inspection of underwater structures by computing optimal trajectories. In scientific computing, FlashDiffusion leverages fused GPU tiles for spectral decomposition, and new GPU-based evaluators are speeding up formal theorem proving, processing 3.5 proof candidates per second on an NVIDIA H100 GPU.

Related