TERRA framework tackles memory demands for high-resolution AI Earth modeling

Researchers have developed TERRA, a new framework designed to handle the memory-intensive demands of training high-resolution AI-based Earth forecasting models. TERRA introduces Sampling-Aware Window, Sequence, and Tensor Parallelism (SAWSTP) to manage attention costs and memory usage, particularly for models like Swin Transformers. The framework also incorporates Memory Orchestration (MO) for efficient checkpoint planning and activation offloading during long-lead finetuning. Experiments demonstrate TERRA's capability to train models with up to 11.4 billion parameters on 96 H200 GPUs, achieving significant scaling efficiencies and reducing peak GPU memory usage. AI

IMPACT Enables training of larger, more accurate AI models for Earth forecasting by optimizing memory and computational efficiency.

RANK_REASON The cluster describes a new framework and parallelism techniques presented in an academic paper for AI-based Earth modeling. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CV →

• GLORYS
• H200 GPUs
• Sampling-Aware Window, Sequence, and Tensor Parallelism
• Swin Transformers

• infra
• paper
• model release

AI-generated summary · Google Gemini · from 1 sources. How we write summaries →