Explain data partitioning and shuffling in Spark. Why are shuffles expensive, and how can you minimize them?

Sample answer preview

Partitioning and shuffling are central to Spark performance. Understanding how data is distributed across the cluster and when redistribution happens is essential for writing efficient jobs. Partitioning determines how data is divided across cluster nodes.