What are the common failure modes in distributed training, and how do you debug them? Discuss issues like gradient synchronization failures, memory errors, and convergence problems at scale.

Sample answer preview

Distributed training introduces failure modes that do not exist in single-device training. Effective debugging requires understanding common issues, their symptoms, and systematic approaches to diagnosis. Gradient synchronization failures cause training to stall or crash.