A news summary based on the article body: Distributed training for large models faces two problems: time and space. DDP (Distributed Data Parallel) attacks time by having every GPU keep a full copy of the model and train on a different slice of the batch. FSDP (Fully Sharded Data Parallel) attacks space by splitting the model so no GPU holds the entire model. Between these sit the ZeRO stages from DeepSpeed, which let users slide from one end to the other step by step. All strategies work by moving data between GPUs, and speed depends on the physical wires connecting them. The same code on the same kind of GPUs ran several times slower on one node than on another. Strategy comparisons ran on four A100 80GB GPUs; hardware experiments ran on H200 machines, including two with the exact same GPUs wired together in different ways.