TACO framework boosts LLM training throughput by 1.87X with tensor compression

Researchers have introduced TACO, a novel framework designed to enhance the efficiency of training large-scale tensor-parallel Large Language Models (LLMs). TACO addresses communication overhead by employing an FP8-based compression strategy for intermediate tensors, utilizing data-driven reshaping and an Adaptive Scale-Hadamard Transform for high-fidelity quantization. The framework also features a fused compression operator to reduce memory traffic and kernel launch times, enabling better overlap with communication. Experiments with GPT and Qwen models showed TACO can improve end-to-end throughput by up to 1.87 times with minimal accuracy loss. AI

影响 TACO's efficiency gains could accelerate large-scale LLM training, potentially lowering compute costs and enabling faster iteration cycles.

排序理由 This is a research paper detailing a new method for LLM training efficiency.

在 arXiv cs.AI 阅读 →

• GPT
• LLM
• Qwen
• TACO
• FP8

• 论文
• 基础设施

AI 生成摘要 · Google Gemini · 来自 1 个来源。 我们如何撰写摘要 →