What techniques reduce communication overhead in distributed training? Explain gradient compression, communication-computation overlap, and hierarchical communication strategies.

Sample answer preview

Communication overhead often limits distributed training scalability, especially as device counts increase. Optimizing communication is essential for achieving near-linear scaling at large cluster sizes.