vLLM & TGI Guide: Scaling Quantized LLMs on Multi-GPU Nodes

This guide details a production-grade pipeline for serving quantized open-weight large language models (LLMs) at enterprise scale. It focuses on using vLLM and Text Generation Inference (TGI) with tensor parallelism and paged-attention KV cache on multi-GPU nodes, specifically highlighting NVIDIA H100 GPUs. The guide covers various quantization paths like AWQ, GPTQ, and FP8, and discusses system configurations for achieving sub-100 ms inference latency at 150 TPS. AI

IMPACT Optimizes inference performance for open-weight LLMs, potentially lowering costs and increasing accessibility for enterprise deployments.

RANK_REASON Guide on deploying and optimizing existing LLM serving frameworks.

Read on dev.to — LLM tag →

• FP8
• GPTQ
• llama
• NVIDIA H100
• Open-weight LLMs
• PagedAttention
• Tensor Parallelism
• Text Generation Inference
• vLLM

• infra
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →