32B scale

Untied Ulysses enables large-scale LLM training with 3M token context

Training large language models with extensive context windows, such as 3 million tokens, faces memory limitations on hardware like 8xH100 nodes. Researchers have developed a method called Untied Ulysses to overcome thes…