Ollama production setup details GPU memory management and load balancing

This post details a production setup for Ollama, focusing on managing GPU memory and concurrent load. The author describes a hybrid strategy for GPU residency, pinning frequently used models like qwen3:8b and BGE M3-Embedding in memory while running less common models on a separate instance. They emphasize tuning Ollama's `OLLAMA_MAX_LOADED_MODELS` parameter to prevent VRAM thrashing, citing a production incident where latency spiked significantly. The architecture utilizes Kong API Gateway for load distribution across multiple Ollama instances, routing requests to specialized services like NER and summarization. AI

IMPACT Provides practical insights for deploying and managing LLM inference at scale, particularly concerning GPU resource optimization and load balancing strategies.

RANK_REASON The article describes a specific technical setup and operational strategy for using an existing tool (Ollama) in a production environment, rather than a new release or significant industry event.

Read on dev.to — LLM tag →

• BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation
• Celery
• customRatnaDveLinga
• gemma3:4b
• Kong Inc.
• llama3.2
• Ollama

• infra
• product

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →