FMplex: Model Virtualization for Serving Extensible Foundation Models
Researchers have developed FMplex, a novel system designed to optimize the serving of foundation models (FMs) by treating them as a virtualization substrate. This approach allows multiple downstream tasks to share a single physical FM instance, reducing memory waste and amortizing costs associated with batching and loading. FMplex enables task-specific extensions and isolation while improving efficiency, demonstrated by significant reductions in latency and increased task hosting capacity. AI
IMPACT Optimizes foundation model deployment, potentially reducing infrastructure costs and improving latency for AI applications.