Tensor Parallelism
PulseAugur coverage of Tensor Parallelism — every cluster mentioning Tensor Parallelism across labs, papers, and developer communities, ranked by signal.
2 day(s) with sentiment data
-
SwiftQK optimizes LLM training with efficient tensor parallelism
Researchers have developed SwiftQK, a new method for optimizing Query-Key Normalization (QK-Norm) in large-language models trained with Tensor Parallelism. This technique significantly reduces the communication overhead…
-
PyTorch DDP Explained: Gradient Synchronization for Multi-GPU Training
This article provides a deep dive into PyTorch's DistributedDataParallel (DDP) for multi-GPU training. It explains the necessity of DDP due to growing model and dataset sizes, contrasting it with model parallelism. The …
-
New CTA-pipelining method slashes multi-GPU latency for LLMs
Researchers have introduced CTA-pipelining, a novel execution paradigm for multi-GPU systems that optimizes for latency in serving large language models. This method exploits dependencies at the Cooperative Thread Array…
-
STAGE framework synthesizes LLM execution graphs for distributed workloads · 2 sources tracked
A new framework called STAGE has been developed to synthesize high-fidelity execution graphs for large language models (LLMs) and Mixture-of-Experts (MoEs). This framework aims to optimize distributed AI workloads by mo…
-
User seeks advice on dual GPU VRAM upgrade for LLMs amid PCIe concerns
A user on Reddit's r/LocalLLaMA subreddit is seeking advice on adding a second AMD 7900XTX GPU to their system to increase VRAM for local large language model (LLM) inference. The primary concern is the potential perfor…
-
AI verification scales with new parallelism techniques
Researchers have adapted tensor parallelism and fully sharded data parallelism techniques, typically used for training large models, to improve the scalability of neural network verification. These methods address the G…
-
Zyphra's TSP strategy boosts LLM training throughput by 2.6x
Zyphra has developed a new technique called Tensor and Sequence Parallelism (TSP) designed to optimize the training and inference of large transformer models. This hardware-aware strategy combines aspects of Tensor Para…
-
New TSP strategy folds tensor and sequence parallelism for memory-efficient training
Researchers have introduced a new parallel execution strategy called Tensor and Sequence Parallelism (TSP) designed to enhance memory efficiency during the training and inference of Transformer models. TSP combines tens…