How do you design a training system for trillion-parameter models using 3D parallelism? Explain how to combine data, pipeline, and tensor parallelism, and how to determine the optimal configuration.

Sample answer preview

Training trillion-parameter models requires combining multiple parallelism strategies, each addressing different constraints. 3D parallelism orchestrates data, pipeline, and tensor parallelism in a unified system.