Introduction & Network Collectives
---
Understanding DataLoaders in DDP
DataLoader Diagram
Anatomy of a DDP Training Script
We'll build a DDP pre-training script step by step, evolving from a single-GPU baseline to a full multi-node script using Qwen-2.5-0.5B as the model.
Running on SageMaker HyperPod with Slurm
Cluster Assumptions
Running on SageMaker HyperPod with EKS
Cluster Assumptions
Distributed Checkpointing
In multi-node training, checkpointing is critical — jobs can be preempted, nodes can fail, and long-running pre-training runs must be resumable. DDP checkpointing has a few subtleties you need to handle correctly.
What's Next
Gradient Accumulation with no_sync()