The Clustering Strikes Back: Building Cost-Effective and High-Performance ANNS at Scale with Helmsman
Researchers at RedNote (Xiaohongshu) have developed HELMSMAN, a new clustering-based approximate nearest neighbor search (ANNS) system designed to significantly reduce hardware costs for large-scale ANNS deployments. By integrating a userspace storage stack, a learned pruning module, and GPU-accelerated construction pipelines, HELMSMAN achieves substantial savings, reducing hardware costs by over 90%. The system can handle billion-scale index rebuilds within hours and currently supports ANNS workloads on 40 machines that previously required approximately 35,000 cores and 0.35 PB of DRAM. AI
IMPACT Reduces hardware costs for large-scale ANNS, potentially enabling wider adoption of AI-powered search and recommendation systems.