Distributed Training

Distributed training is the practice of training a machine-learning model across multiple GPUs or machines rather than a single device, splitting either the data, the model's parameters, or the training pipeline between them. It is the technique that makes training of modern large language models possible: models with hundreds of billions of parameters and training corpora of trillions of tokens cannot fit in the memory of one accelerator, nor finish training in a reasonable time on one.^[1]

Parallelism Strategies

Data parallelism replicates the model on every accelerator, feeds each a different mini-batch, and synchronises gradients after each step — typically with an all-reduce operation. It is simple and scales well but duplicates the full model and optimizer state on every device.

Sharded data parallelism attacks that memory cost. Microsoft's ZeRO (Zero Redundancy Optimizer) partitions optimizer states, gradients and parameters across devices, so each machine stores only a fraction of the redundant data while retaining data-parallel communication patterns; PyTorch's FSDP (Fully Sharded Data Parallel) implements the same idea.^[2]

Tensor (model) parallelism splits individual weight matrices across devices so that a single layer's computation is shared — the approach popularised by NVIDIA's Megatron-LM — allowing a model too large for one GPU to be run layer-parallel across several.^[3]

Pipeline parallelism cuts the model into sequential stages (embedding, transformer blocks, output head) assigned to different devices, streaming micro-batches through the "pipeline". Real systems combine all three, producing 3D parallelism configurations for frontier models.

The Communication Bottleneck

The limiting factor is not raw compute but interconnect bandwidth: gradients and activations must cross the network between steps, and slow links stall every GPU in the cluster. High-speed fabrics such as InfiniBand with RDMA, NVLink within a node, and topology-aware scheduling are what separate efficient clusters from expensive ones. Practical training also relies on mixed-precision arithmetic, gradient accumulation, activation checkpointing and frequent checkpointing to survive multi-week runs.^[4]

Scale is staggering: BLOOM's 176B parameters were trained in 2022 on 384 A100 GPUs using Megatron-DeepSpeed, and Meta's Llama 3 405B model used a cluster of roughly 16,000 H100 GPUs.^[5] Efficiency is measured as scaling efficiency — how much throughput a cluster retains as it grows — and diminishing returns from communication overhead are a major driver of hardware and software design in the AI infrastructure industry.

>See Also

• GPU Clusters
• DeepSpeed
• Ray
• InfiniBand
• Mixed-Precision Training
• MLOps