When and how should you use caching and persistence in Spark? What are the different storage levels, and what are the trade-offs?

Sample answer preview

Caching and persistence allow Spark to store intermediate results in memory or on disk so they can be reused without recomputation. Used correctly, they dramatically improve performance. Used incorrectly, they waste memory and can actually slow jobs down.