Datasets: Type-Safe Structured APIs

The Dataset API is a type-safe version of Spark's Structured API, for writing statically typed code in Java and Scala. It is not available in Python or R, because those languages are dynamically typed.

Recall that a DataFrame is a distributed collection of objects of type Row, which can hold many kinds of tabular data. The Dataset API lets you assign a Java/Scala class to the records inside a DataFrame and manipulate them as a collection of typed objects — much like a Java ArrayList or a Scala Seq.

The APIs on a Dataset are type-safe: you cannot accidentally view its objects as a different class than the one you put in initially. That safety makes Datasets especially attractive for writing large applications where multiple engineers must interact through well-defined interfaces.