A developer has created a new PyTorch optimizer called ApexW, designed to reduce memory usage and speed up optimizer steps for large GPU workloads. Early benchmarks show ApexW can be up to 1.5x faster than PyTorch's fused AdamW and cut optimizer-state memory usage in half, particularly for models with large tensors. However, it faces challenges with workloads involving many small tensors due to Python loop overhead and does not yet offer significant end-to-end training speedups over PyTorch's integrated C++ implementation. AI
IMPACT May enable larger models or batch sizes on existing hardware by reducing optimizer memory footprint.
RANK_REASON Developer-created tool release, not from a major lab.
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →
