Brief

Researchers have developed a new memory management technique called Asymmetric Virtual Memory Paging (AVMP) to improve the efficiency of hybrid language models. These models combine Transformer layers with State Space Models (SSMs), leading to distinct memory cache types that current systems handle poorly. AVMP separates these cache types into distinct pools and allows capacity migration between them when needed, reducing out-of-memory events and significantly boosting request throughput. AI

Researchers have identified and exploited activation subspace bottlenecks within Mamba-family state-space models (SSMs) to improve their performance. By applying a simple scalar multiplication to these bottleneck activations during testing, they achieved an average performance increase of 8.27% across multiple SSMs and benchmarks without task-specific tuning. Further validation through retraining a modified architecture, dubbed Stable-Mamba, demonstrated significant long-context performance gains, confirming the identified bottlenecks' impact on hindering performance. AI