HyTopK

HyScale system accelerates long-context LLM training with sparse attention

Researchers have developed HyScale, a new training system designed to optimize the process of training large language models with long contexts, specifically focusing on learned sparse attention mechanisms. This system …