📄️ Introduction & Network Collectives
---
📄️ Understanding DataLoaders in DDP
DataLoader Diagram
📄️ Anatomy of a DDP Training Script
We'll build a DDP pre-training script step by step, evolving from a single-GPU baseline to a full multi-node script using Qwen-2.5-0.5B as the model.
📄️ Running on SageMaker HyperPod with Slurm
Cluster Assumptions
📄️ Running on SageMaker HyperPod with EKS
Cluster Assumptions
📄️ Distributed Checkpointing
In multi-node training, checkpointing is critical — jobs can be preempted, nodes can fail, and long-running pre-training runs must be resumable. DDP checkpointing has a few subtleties you need to handle correctly.
📄️ What's Next
Gradient Accumulation with no_sync()