Luma Labs 发布了其在高效多模态序列处理方面（尤其是在打包序列方面）的进展细节。他们开发了提高性能和降低计算开销的技术，例如稀疏全连接获取机制和保持 FP32 精度的在线 FlashAttention-4 内核。这些创新实现了更快的处理速度和更好的扩展性，他们的新方法在大多数测试场景中均优于 FlashMask 和 Ulysses 等现有方法。 AI
影响 序列并行和注意力机制方面的这些创新可能导致多模态 AI 模型更高效的训练和推理。
排序理由 该集群详细介绍了 Luma Labs 新 AI 处理技术的技术研究和实现细节，包括特定的内核优化和注意力机制。
在 X — Luma Labs (video gen) 阅读 →
AI 生成摘要 · Google Gemini · 来自 7 个来源。 我们如何撰写摘要 →
完整方法见我们的编辑标准。
We're at PyTorch Conference this week. Come talk distributed training and kernels at our happy hour, Tue Oct 20 in San Jose: https://t.co/cCwXfUhljE
The margin grows with scale: a median of 1.28× over FlashMask + Ulysses at 32 GPUs and 2.63× at 128. Full write-up: https://t.co/VFujPDSDgE https://t.co/9E0QGM3kUA
When queries see only a few samples, a sparse all-to-all fetches just the key/value blocks each rank needs, 55–100% fewer remote keys than ring attention on our sequences. For a sliding window, that is only the last window. https://t.co/ftgNH5d528
Standard ring attention rounds partial outputs to bf16 every step. Our online FlashAttention-4 kernel keeps the softmax state in fp32 across ring steps, so outputs and gradients match a single GPU. https://t.co/fTfQmImaT8
Ring attention waits for its slowest rank at every step. We balance it with PTRR from PyTorch's context parallelism, applied to block-sparse costs: the busiest rank lands within 0.1% of the rest and the layer time roughly halves. https://t.co/BZk79GeSS8
A packed sequence mixes causal text, bidirectional images and video, generated frames that only read earlier clean frames, and sliding windows. We plan all of it from per-token metadata and never build the full mask. https://t.co/rMXthCPV3a
Exact sequence parallelism for packed multimodal masks. It picks ring attention, a sparse all-to-all or Ulysses per layer, matches single-GPU numerics, and beats a FlashMask + Ulysses hybrid in 47 of 48 cases on up to 128 H100s. https://t.co/VFujPDSDgE https://t.co/jfpzATvJn5