Data Parallelism

Data parallelism is the most common distributed-training strategy, in which each GPU holds a full copy of the model and processes a different shard of the training batch. After each step the devices synchronize gradients with an all-reduce so every replica stays identical. It scales throughput nearly linearly when interconnect bandwidth keeps up, and is often combined with tensor and pipeline parallelism in large clusters.