Lazy Evaluation

Lazy evaluation means Spark waits until the very last moment to execute the graph of computation instructions. Instead of modifying the data immediately when you express an operation, you build up a plan of transformations to apply to your source data.

By waiting until the last minute, Spark compiles this plan from your raw DataFrame transformations into a streamlined physical plan that runs as efficiently as possible across the cluster. This lets Spark optimize the entire data flow end to end.