Gain insights into Spark, its architecture, application lifecycle, and APIs. Delve into data frames, datasets, and Spark SQL to effectively manage and query big data.

spark.tar.gz

SparkShellUI

SparkHistoryServerUI

Spark has come to dominate the big data processing space in a short span of time since its release and now serves as the de-facto unified big data processing engine in the industry. 

In this course, you will get a complete introduction to the basics of Spark. You will start by learning about the architecture, the application lifecycle, and its API.

From there, you will dive into the data frame data structure and its API as well as the strongly-typed datasets API. Lastly, you’ll get into the Spark SQL engine which will allow you to issue queries on structured data with a schema.

By the end of this course, you will have the confidence to use Spark in any of your big data projects.

An Introduction to Spark

# Datasets
<p>Below is the definition of a Dataset from the official Databricks documentation: 

> *"A Dataset is a strongly-typed, immutable collection of objects that are mapped to a relational schema. Datasets are a type-safe structured API available in statically typed, Spark supported languages Java and Scala. Datasets are strictly a JVM language feature. Datasets aren't supported in R and Python since these languages are dynamically typed languages".*

 After Spark 2.0, RDD was replaced by Dataset, which is strongly-typed like an RDD, but with richer optimizations under the hood.

	







In the context of Scala we can think of a DataFrame as an alias for a collection of generic objects represented as `Dataset[Row]`. The `Row` object is untyped and is a generic JVM object that can hold different types of fields. In contrast, a Dataset is a collection of strongly typed JVM objects in Scala or a class in Java. It is fair to say that each Dataset in Scala has an untyped view called DataFrame, which is a Dataset of `Row`. The following table captures the notion of Datasets and DataFrames in various Spark supported languages. 



| Language  |  Typed and untyped main abstraction | Typed or Untyped  |
| - | - | - |
|  Scala | Dataset[T] and DataFrame  | Both typed and untyped  |
|  Java | Dataset<T>  | Typed  |
|  Python | DataFrame  | Generic Row untyped  |
|  R | DataFrame  | Generic Row untyped  |


Datasets are possible because of a feature called the encoder, which converts from JVM types to Spark SQL’s specialized internal (tabular) representation. Encoders are highly specialized and optimized code generators that generate custom bytecode for the serialization and deserialization of your data. Encoders are required by all Datasets. The encoder maps domain-specific types to Spark’s internal representation for that type. For instance, an `Int` that is one of the fields in a `Row` will be mapped or converted to `IntegerType` or `IntegerType()` respectively for Scala or Java and Python. Domain-specific types can be expressed as beans for Java and as case classes for Scala. The tabular representation is stored using Spark’s internal Tungsten binary format, allowing for operations on serialized data and improved memory utilization. Whilst using the Dataset API, Spark generates code at runtime to serialize a Java object into an internal binary structure and vice versa. This conversion can have a slight impact on performance, but there are several benefits to be had. For example, Spark understands the structure of data in Datasets, and it can create a more optimal layout in memory when caching Datasets.


As an example, consider the following snippet of code where we instantiate a `Row` object in the Scala shell which is untyped. Later, we attempt to retrieve the different fields of the `Row` object using getter methods and passing in the field index. Note that when we attempt to read a string as an integer or vice versa, the cast fails. The `row` variable itself is an untyped generic object. If we were using a Dataset, then each element in the Dataset would map to a JVM object.

```shell
scala> var row = Row("Upcoming New Movie", 2021, "Comedy")
row: org.apache.spark.sql.Row = [Upcoming New Movie,2021,Comedy]

scala> row.getInt(0)
java.lang.ClassCastException: java.lang.String cannot be cast to java.lang.Integer
  at scala.runtime.BoxesRunTime.unboxToInt(BoxesRunTime.java:103)
  at org.apache.spark.sql.Row.getInt(Row.scala:245)
  at org.apache.spark.sql.Row.getInt$(Row.scala:245)
  at org.apache.spark.sql.catalyst.expressions.GenericRow.getInt(rows.scala:166)
  ... 47 elided

scala> row.getInt(1)
res125: Int = 2021

scala> row.getString(0)
res126: String = Upcoming New Movie

scala> row.getString(1)
java.lang.ClassCastException: java.lang.Integer cannot be cast to java.lang.String
  at org.apache.spark.sql.Row.getString(Row.scala:277)
  at org.apache.spark.sql.Row.getString$(Row.scala:277)
  at org.apache.spark.sql.catalyst.expressions.GenericRow.getString(rows.scala:166)
  ... 47 elided
```

# Differences with DataFrames
<p> We'll need to contrast DataFrames and Datasets to gain a better understanding of the two. Datasets check if types conform to the specification at compile time. DataFrames aren't truly untyped, as their types are maintained by Spark, but the verifying whether the types conform to the specification in the schema is done at runtime. Said another way, DataFrames can be thought of as Datasets of type `Row`, which is Spark's internal optimized in-memory representation for computation. Having its own internal representation of types allows Spark to skip JVM types that can be slow to instantiate and have garbage collection costs. We'll study this aspect of Spark in later lessons.</p>

# Use-cases for Datasets

It may seem redundant to have Datasets when we already have DataFrames, but there are certain scenarios in which Datasets are a better fit than DataFrames. That is, some operations can't be expressed with DataFrames but only with Datasets. Consider also the desire for type-safety. For example, attempting to multiply two string variables in code will fail at compile-time instead of at run-time. Additionally, development may be helpful as IDEs and other tools can provide auto-complete and other hints when objects are strongly typed. Another reason to use Datasets is if all of one's data and transformations accept case classes (Scala); it is trivial to reuse them for both distributed and local workloads.




Get an introduction to the strongly typed Datasets API available in Spark.

Spark Overview

DataFrames

Datasets

Spark SQL

Summary

Datasets

Datasets