What are the main parallelism strategies in distributed deep learning training, and when would you use each approach?

Question

Accepted Answer

Distributed training accelerates model training by leveraging multiple compute devices. Three main parallelism strategies address different bottlenecks: data parallelism for throughput, model parallelism for memory, and pipeline parallelism for efficiency with large sequential models.

Data parallelism is the most common approach. The model is replicated across all devices, and the training data is split into shards. Each device processes a different data batch using its local model copy. After the backward pass, gradients are synchronized across all devices using an all-reduce operation, ensuring all model copies remain identical. Data parallelism scales well and is straightforward to implement when the model fits in a single device's memory.

Model parallelism splits the model itself across devices rather than replicating it. Tensor parallelism divides individual layers across devices, with each device computing part of the layer's output. This is useful for very wide layers that exceed single-device memory. The trade-off is increased communication since intermediate activations must be exchanged between devices during both forward and backward passes.

Pipeline parallelism partitions the model vertically by layers, assigning different layer groups to different devices. Data flows through the pipeline as micro-batches. While one device processes the forward pass for micro-batch N, the next device can work on micro-batch N-1. This approach reduces memory per device and can achieve good hardware utilization with careful scheduling, though it introduces pipeline bubbles where devices wait for their inputs.

Choosing the right strategy depends on your constraints.

Use data parallelism when the model fits in single-device memory and you want to increase throughput. This is the default choice for most training scenarios.

Use model parallelism when individual layers are too large for single-device memory, common with massive embedding tables or very wide attention layers.

Use pipeline parallelism when the full model exceeds device memory but can be partitioned cleanly into stages. This is particularly effective for deep transformer models.

Modern large-scale training often combines all three approaches in what is called 3D parallelism, using data parallelism across nodes, pipeline parallelism within nodes, and tensor parallelism within each pipeline stage. This combination enables training models with hundreds of billions of parameters across thousands of devices.