vLLM: Running Two AI Models Simultaneously on a Single GPU

A technical guide demonstrates how to run two separate vLLM servers on a single GPU, specifically an RTX 3090, by carefully managing memory utilization. The author found that starting a chat model server with a high memory allocation (0.9) prevented a smaller embedding model server from starting due to vLLM's startup memory check. The solution involves assigning a very low memory utilization fraction (0.03) to the embedding model, which uses significantly less memory than allocated. Running both models concurrently resulted in a performance decrease of approximately 8-12% for each server. AI

IMPACT Provides a practical solution for users looking to maximize GPU utilization for running multiple AI models concurrently.

RANK_REASON The item describes a technical workaround for optimizing resource usage with existing software, rather than a new release or significant industry development.

Read on dev.to — LLM tag →

• BAAI/bge-small-en-v1.5
• cuda:0
• qaisermehdi3-coder/qvunex
• Qwen2.5-1.5B-Instruct
• RTX 3090
• vLLM
• vllm-project/vllm

• infra

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →