Explain mixed precision training and its benefits for distributed training. How does loss scaling prevent underflow, and what precision choices are appropriate for different operations?

Sample answer preview

Mixed precision training uses lower precision arithmetic for most operations while maintaining higher precision where necessary for numerical stability. This approach significantly accelerates training and reduces memory usage, both critical for distributed training at scale.