Multi-GPU Parallelism Techniques

📖 The A-to-Z of Distributed Training Parallelism • http://parallelism.aman.ai - Distributed training parallelism is crucial for efficiently training large-scale deep learning models that require extensive computational resources. This approach leverages multiple GPUs or machines to perform computations in parallel, significantly reducing training time and enabling the handling of larger datasets and models. - There are four main strategies for parallelism in distributed training: model, data, pipeline, and tensor parallelism. Each has its own mechanisms, advantages, and challenges, and understanding them is essential for optimizing training performance in different scenarios. 🔹 Types of Parallelism (Data, Model, Pipeline, Tensor) 🔹 Choosing the Right Strategy: Data v/s Model v/s Pipeline v/s Tensor Parallelism 🔹 Data Parallelism - DataParallel (DP) (How DataParallel Works, Key Steps, Code) - Distributed Data Parallel (DDP) (Key Features of DDP, Steps to Use DDP, Code) 🔹 Model Parallelism - Layer-wise Parallelism - Tensor-wise Parallelism - Operator-wise Parallelism - Comparative Analysis: Types of Model Parallelism 🔹 Hybrid (Model and Data) Parallelism - Fully Sharded Data Parallel (FSDP) (Key Features of FSDP, Technical Details, Code + Explanation) - Benefits of FSDP 🔹 Tensor Parallelism (Concept, Mechanism, Types of Tensor Parallelism, Pros and Cons, Use Cases, Implementation in PyTorch, Conclusion) 🔹 Pipeline Parallelism (Concept, Mechanism, Types of Pipeline Parallelism, Pros and Cons, Use Cases, Implementation in PyTorch, Conclusion) 🔹 DeepSpeed (Key Features of DeepSpeed, Technical Details, Code + Explanation, Benefits of DeepSpeed) 🔹 DeepSpeed ZeRO (Key Features of DeepSpeed ZeRO, Technical Details, Benefits, Code + Explanation, Comparison of ZeRO Stages) #artificialintelligence #genai #deeplearning #neuralnetworks

I just came across a fascinating paper titled "FlexSP: Accelerating Large Language Model Training via Flexible Sequence Parallelism" that presents an innovative approach to improving the efficiency of LLM training. The Challenge: Training LLMs with long sequences is incredibly resource-intensive. Traditional sequence parallelism methods assume all input sequences are the same length. In reality, training datasets have a wide, long-tail distribution of sequence lengths. This mismatch leads to load imbalance—some GPUs finish early while others lag behind on longer sequences, causing inefficiencies and wasted throughput. The FlexSP Solution: FlexSP introduces an adaptive, heterogeneity-aware sequence parallelism strategy. Instead of using a fixed partitioning strategy, FlexSP dynamically adjusts how sequences are divided across GPUs for each training step. It does this by: Forming Heterogeneous SP Groups: Allocating larger parallelism groups to process long sequences (to avoid out-of-memory errors) and smaller groups for short sequences (to minimize communication overhead). Time-Balanced Sequence Assignment: Solving an optimization problem (via a Mixed-Integer Linear Program enhanced with dynamic programming for bucketing) to balance the workload across GPUs and reduce idle time. Key Benefits: Significant Speedups: The adaptive approach can achieve up to a 1.98× speedup compared to state-of-the-art training frameworks, effectively cutting down training time. Improved Resource Utilization: By intelligently adapting to the heterogeneous nature of real-world datasets, FlexSP ensures that all GPUs are utilized efficiently, regardless of sequence length variation. Scalability: The system is designed to work with current distributed training systems and can seamlessly integrate with other parallelism strategies. This paper is a brilliant example of how rethinking parallelism to account for real-world data variability can lead to substantial performance improvements in training large language models. If you’re interested in the future of LLM training and efficient GPU utilization, I highly recommend giving FlexSP a read. Wang, Y., Wang, S., Zhu, S., Fu, F., Liu, X., Xiao, X., Li, H., Li, J., Wu, F. and Cui, B., 2024. Data-Centric and Heterogeneity-Adaptive Sequence Parallelism for Efficient LLM Training. arXiv preprint arXiv:2412.01523. #LLM #DeepLearning #AI #GPU #Parallelism #MachineLearning #TrainingEfficiency #FlexSP

Serving DeepSeek-R1's 671B Parameters Across a Multi-Node Nvidia H100 Cluster TL;DR 🧠 DeepSeek-R1 671B parameters exceed single-node 8xH100 node VRAM – let alone the additional memory for activations and KV cache. 🔗 Multi-node inference solves this by using 16 H100 GPUs across two nodes (1280GB total VRAM), leveraging expert parallelism to distribute the model efficiently. ⚡ During inference, large models rely on VRAM to store and read massive amounts of weights (e.g., 37GB for a single 37B active-parameter expert at fp8 precision for DeepSeek-R1).  🛠️ VRAM bandwidth (3.35 TB/s on H100 GPUs) is primary inference bottleneck for DeepSeek-R1 given that each forward pass transfers 37 GBs of data directly from VRAM for each expert 📊 Must parallelize model inference across nodes and GPUs for efficient performance across nodes 🖥️ Requires hardware-topology-aware design and efficient parallelism (expert parallelism) to increase inference performance. Problems & Solutions 🔧 Problem: DeepSeek-R1’s 671B parameters + KV cache/activations exceed the VRAM of a single 8xH100 node (640GB). 🛠️ Solution: Use two 8xH100 nodes (16 GPUs total) with 1280GB VRAM, connected via InfiniBand. 🌐 Problem: Inter-node communication (InfiniBand: 800GB/s) is slower than intra-node GPU communication (NVLink: 900GB/s) and much slower than VRAM bandwidth (3.35 TB/s) 📡 Solution: Reduce slow inter-node data transfer by leveraging expert parallelism, which reduces inter-node communication, utilizes faster VRAM bandwidth, and improves overall inference performance. Novel Insights and Learnings 1️⃣ Interconnects are not the bottleneck: Counter-intuitively, VRAM bandwidth (3.35TB/s) is the primary bottleneck due to the amount of data that is loaded from VRAM on each inference request  2️⃣ Expert parallelism maps experts to GPUs based on physical interconnects - reducing cross-node traffic, doing more work within each node (using more VRAM bandwidth), and improving overall inference performance. 3️⃣ While tensor parallelism puts an equal portion of each expert in each GPU, expert parallelism puts an equal number of whole experts in each GPU. Future Work 🧭 Scaling to 1024 GPUs: Testing 128-node clusters for trillion-parameter models. 🔄 Dynamic Parallelism: Switching between tensor and expert parallelism mid-inference for adaptive workloads. Key Visualizations 📊 Figures 1 & 2: DeepSeek-R1 671B parameters exceed single-node 8xH100 node VRAM – let alone the additional memory for activations and KV cache. 📡 Figure 3: DeepSeek-R1 runs on two 8xH100 nodes (1280GB total VRAM) connected via InfiniBand 🔗 Figure 4: Tensor Parallelism vs. Expert Parallelism - each GPU holds 16 experts, distributing the 256 experts across the 16 GPUs for efficient inference. 📰 Blog: How multi-node inference works for massive LLMs like DeepSeek-R1 (https://lnkd.in/gCPM58NB) by Philip Kiely and Philip Howes

عند بداية استخدام التدريب الموزع في #ML، تُعتبر تقنية "توازي البيانات" (Data Parallelism) مدخلًا بسيطًا وفعّالًا. تقوم هذه الطريقة على توزيع البيانات على GPUs بينما تحتفظ كل GPU بنسخة كاملة من النموذج. كل GPU تعالج جزء من البيانات بشكل مستقل، ثم تُجمّع gradients باستخدام عملية all-reduce لتحديث الأوزان. الميزة الكبرى؟ البساطة! معظم أطر التعلم الحديثة مثل PyTorch وJAX توفر APIs لتسهيل إدارة الاتصال والتنسيق بين الأجهزة. والأفضل من ذلك، أن تعديل كود التدريب ليعمل بتوازي البيانات غالبًا لا يتطلب تغييرات جذرية — مجرد تغليف للنموذج، واستخدام data loader موزع، وتحديد الأجهزة المشاركة. لكن هناك حدود. يجب أن يتسع النموذج بالكامل في ذاكرة كل GPU. ومع زيادة عدد الـGPUs، قد تُصبح عملية تجميع gradients عنق زجاجة بسبب قيود الشبكة. كما أن توازي البيانات يتطلب تزامنًا تامًا بين الأجهزة، ما يعني أن أي GPU بطيء سيؤخر الجميع — خاصة في بيئات غير متجانسة. القاعدة الذهبية هنا؟ قلل من أوقات خمول الـGPU قدر الإمكان، ويفضل استخدام hardware متماثل لضمان الكفاءة. --- For teams just starting out with distributed training, data parallelism is a straightforward approach to consider. In this strategy, the input data are split across multiple GPUs while maintaining a full copy of the model on each device, allowing for faster training and larger batch sizes. Each GPU processes its portion of the data independently and calculates gradients. These gradients are then aggregated using an all-reduce operation to update the model weights synchronously. The key advantage of data parallelism is its simplicity: most modern deep learning frameworks, such as PyTorch and JAX, provide high-level APIs to handle the inter-device communication and orchestration necessary to coordinate their work. Data parallelism often requires minimal modifications to an existing single-GPU training script: you mainly need to wrap the model, use a distributed data loader, and handle the initialization of the participating devices (e.g., a process group in PyTorch specifies where the model replicas would be loaded). The main limitation is that the entire model, along with the memory required for training it, must fit in the memory of each GPU. There is also the communication overhead that arises from all-reduce operations. As the number of GPUs increases, gradient aggregation can become a bottleneck, especially if network bandwidth or latency is suboptimal. This overhead can diminish scalability gains and complicate performance tuning. In distributed training and in all settings where GPUs are used, it is desirable to reduce GPU idle time. In data-parallel training, synchronous model updates require all GPUs to finish their work for the current training step before aggregating the gradients. Thus, it is beneficial to distribute the work across relatively uniform hardware (e.g., identical GPUs). In heterogeneous environments, where some GPUs or network devices are faster than others, data parallelism may lead to load imbalance — GPUs become idle, waiting for others to finish each training step.

🚀 Need to serve millions of tokens in real time? Meet Helix Parallelism. Problem: Long‑context LLMs choke on two bottlenecks—📚 KV‑cache streaming that hammers DRAM bandwidth, and 🏋️ FFN weight reads that stall every token. Solution: Helix Parallelism, co‑designed for the Blackwell architecture + FP4 compute, stitches together KV, Tensor, and Expert parallelism in a “DNA‑helix” execution flow. One GPU pool is reused for both attention and FFN, eliminating idle time. Impact: Simulations on GB200 NVL72 show up to 32× more concurrent users at the same latency and up to 1.5× lower token‑to‑token latency for low‑concurrency cases. All while pushing the throughput‑latency Pareto frontier for multi‑million‑token decoding. https://lnkd.in/g8vhr7s3 #GenerativeAI #LLM #Inference #NVIDIA #Blackwell #HelixParallelism #SystemsEngineering

Pipeline Parallelism in a Nutshell Think of training a big model (multi-billion parameters) like a three-person kitchen: one friend prepares ingredients, one cooks, one plates the food. Instead of waiting for an entire dish to finish before starting the next, each friend keeps their hands busy on a new batch as soon as the previous one moves down the line. This is pipeline parallelism in a nutshell; splitting the model into stages so while the first GPU is still computing layer one of batch 1, the second GPU is already computing layer two of the same batch and layer one of batch 2. The key here is while ZeRO does sharding of the model parameters, optimizer states, and such, pipeline parallelism just allocates everything per gpu. This means that there is essentially n optimizers where `n` is the chunks of model that we split across the GPUs (when training). The win is hiding the slow stuff. Forward passes are quick math, but shuffling huge weight matrices between GPUs is like waiting for the oven to heat. By stacking mini-batches in a pipe, the communication time overlaps with the next batch’s math, so the expensive “oven” never cools. The trick is keeping every slice of the model small and predictable. If one layer suddenly grows or the data arrives unevenly, the whole pipeline backs up like a kitchen where the prep station hits a surprise pile of onions. Keep the chunks bite-sized, balance the load across GPUs, and you’ll squeeze extra speed out of the same hardware without needing the communication overhead that ZeRO gives you. As a result, this also is why pipeline parallelism is considered favorable when serving models: there’s no wait time for model results, as soon as one minibatch is done with a layer chunk it’s immediately fed to the next where finally once it’s through the last layer you take this output and serve it to your users.

Training large-scale models—particularly LLMs with hundreds of billions or even trillions of parameters—poses unique system-level challenges. Memory limits, communication bottlenecks, and uneven compute loads can quickly bring naïve training strategies to a halt. Relying on just one form of parallelism (e.g., data parallelism alone) simply doesn’t scale effectively. Instead, modern deep learning frameworks and teams combine multiple forms of parallelism to stretch hardware capabilities to their limits. Each strategy addresses a different bottleneck: ➜ Data parallelism boosts throughput by replicating the model across nodes. ➜ Tensor/model parallelism breaks up massive weight matrices. ➜ Pipeline parallelism improves utilization across deep architectures. ➜ Expert parallelism adds sparsity and dynamic routing for efficiency. ➜ ZeRO optimizes memory allocation down to optimizer states and gradients. ➜ Context parallelism (a newer strategy) allows for splitting long sequences—critical for LLMs handling multi-thousand-token contexts. This modular, composable approach is the backbone of training breakthroughs seen in models like GPT-4, PaLM, and beyond. Link to the article: https://lnkd.in/gZBF-N2w

Just finished reading the Ultra-Scale Playbook: Training LLMs on GPU clusters -> here is a quick summary! It's an extremely accessible booklet with just the right amount of details for you to dip your toes into the world of massively distributed GPU computing. I was already familiar with all of the material from the book, and I've previosly recorded some videos going through Megatron, DeepSpeed, 3D parallelism, Flash Attention, etc. and my work on llm.c helped me grok all these concept on the very low level (C/CUDA). Despite that I still learned a few things from the book as I had it all in one place, so I could fit it all into my spatio-temporal context window (heh) and draw new conclusions. :) They go into all major parallelism techniques: * Data parallelism (replicate models across GPUs, shard batch dim) * ZeRO 1, 2, 3 (sharding optimizer state, grads, params) * Model parallelism (Megatron, aka "tensor parallelism" or TP) * Sequence parallelism (complementary to TP) * Pipeline parallelism (AFAB, 1F1B, ..., all the way to DeepSeek's DualPipe) * Context parallelism (ring attention, each GPU handles a subset of the context and attn layer logic needs to be modified) * Expert parallelism (relevant to MoEs how do you shard experts across different devices and route tokens to them back & forth) Subsequently they analyze how you can combine the above before digging into high-level GPU architecture overview (streaming multiprocessors, HBM, shared mem, cache...), writing performant kernels (memory coalescing, tiling, control divergence ideas), and mixed precision. The appendix section is also a must read: * 101 on collective operations (reduce/scatter/gather/broadcast) * heuristics for quickly computing FLOPs/token, comms/computation overlap, etc. * profiling On a side note happy to see they cited my Flash Attention blog :) https://lnkd.in/dV_E79g7 If you want to learn more about this space, and you're curious about systems go read it! Booklet: https://lnkd.in/dqdJGi_p

NVIDIA's other strength is the native software that maximally taps the potential of GPUs. If LLM is the OS for intelligence, then TensorRT is the device driver. All practitioners should check out TensorRT-LLM, an upcoming open-source pkg with batteries included: - Highly optimized, fused kernel for FlashAttention. - Tensor parallelism for inference over multiple GPUs connected by NVLink. - In-flight batching: solution to a very practical problem during deployment. Rather than waiting for the whole batch to finish before moving on to the next set of requests, the runtime immediately evicts finished sequences from the batch. - Easy model compilation to optimized FP8 kernels. - Llama-2 gets ~2X speed up on H100. Launch blog: https://lnkd.in/gbNfgjwp TensorRT-LLM early access: https://lnkd.in/grD-yN9x