Structured API Overview

Spark is a distributed programming model in which you specify transformations. Multiple transformations build up a directed acyclic graph (DAG) of instructions. An action begins executing that graph — as a single job — by breaking it down into stages and tasks that run across the cluster. The logical structures you manipulate with transformations and actions are DataFrames and Datasets: to create a new one you call a transformation; to start computation (or convert to native language types) you call an action.

The Structured APIs are a tool for manipulating all sorts of data — from unstructured log files, to semi-structured CSV files, to highly structured Parquet files. They refer to three core types of distributed collection:

• Datasets
• DataFrames
• SQL tables and views

DataFrames and Datasets are distributed, table-like collections with well-defined rows and columns. Each column must have the same number of rows as every other column (you can use null to mark the absence of a value), and each column carries type information that stays consistent for every row in the collection.

To Spark, DataFrames and Datasets represent immutable, lazily evaluated plans that specify what operations to apply to data at some location to generate an output. When you perform an action on a DataFrame, you instruct Spark to perform the actual transformations and return the result — the plan describes how to manipulate rows and columns to compute the result you want.

Schemas

To make those definitions precise, we need schemas — the way you define the types of data stored in a distributed collection. A schema defines the column names and types of a DataFrame. You can define a schema manually, or read it from a data source — the latter is often called schema on read. A schema is built from types, giving Spark a way to specify what lies where.