Gain insights into Spark, its architecture, application lifecycle, and APIs. Delve into data frames, datasets, and Spark SQL to effectively manage and query big data.

spark.tar.gz

SparkShellUI

SparkHistoryServerUI

Spark has come to dominate the big data processing space in a short span of time since its release and now serves as the de-facto unified big data processing engine in the industry. 

In this course, you will get a complete introduction to the basics of Spark. You will start by learning about the architecture, the application lifecycle, and its API.

From there, you will dive into the data frame data structure and its API as well as the strongly-typed datasets API. Lastly, you’ll get into the Spark SQL engine which will allow you to issue queries on structured data with a schema.

By the end of this course, you will have the confidence to use Spark in any of your big data projects.

An Introduction to Spark

## Simple query example
When we execute Spark SQL queries in the spark-shell, we don’t need to explicitly create a `SparkSession` object. We would if we were writing a Spark application that was to be run using `spark-submit`. The `SparkSession` object is provided implicitly by the shell. Read the listing below, which is similar to what we have done in the previous lessons, except for creating the ***temporary view***, which we discuss later.

```shell
scala> val movies = spark.read.format("csv")
                       .option("header", "true")
                       .option("samplingRatio", 0.001)
                       .option("inferSchema", "true")
                       .load("/data/BollywoodMovieDetail.csv")
movies: org.apache.spark.sql.DataFrame = [imdbId: string, title: string ... 8 more fields]

scala> movies.createOrReplaceTempView("tempView")

scala> spark.sql("SELECT title FROM tempView WHERE releaseYear > 2010 ORDER BY title desc")
            .show(3)
+--------------------+
|               title|
+--------------------+
|            Zokkomon|
|   Zindagi Tere Naam|
|Zindagi Na Milegi...|
+--------------------+
only showing top 3 rows
```

The query returns all the movie titles that were released post-2010 and also sorts them in descending order by title. We then print three rows from the result. The `sql()` method returns the results as a DataFrame. Spark SQL offers an <a href="https://en.wikipedia.org/wiki/SQL:2003" target="_blank">ANSI:2003-compliant</a> SQL interface and the query we executed should look and feel similar to a query against a relational database table such as MySQL or PostgreSQL. We can also express the same query using DataFrames API as follows:


```shell
movies.where($"releaseYear" > 2010)
      .sort(desc("title"))
```

## Complex query example

Let’s try a slightly more complex query. Imagine that  we want to list all the movies released post 2010 and at the same time label each movie below average, average, and above average if the `hitFlop` column is less than 5, equal to 5, and higher than 5, respectively. The query and its output are presented below:

# Simple query example
When we execute Spark SQL queries in the spark-shell, we don’t need to explicitly create a `SparkSession` object. We would if we were writing a Spark application that was to be run using `spark-submit`. The `SparkSession` object is provided implicitly by the shell. Read the listing below, which is similar to what we have done in the previous lessons, except for creating the ***temporary view***, which we discuss later.

```shell
scala> val movies = spark.read.format("csv")
                       .option("header", "true")
                       .option("samplingRatio", 0.001)
                       .option("inferSchema", "true")
                       .load("/data/BollywoodMovieDetail.csv")
movies: org.apache.spark.sql.DataFrame = [imdbId: string, title: string ... 8 more fields]

scala> movies.createOrReplaceTempView("tempView")

scala> spark.sql("SELECT title FROM tempView WHERE releaseYear > 2010 ORDER BY title desc")
            .show(3)
+--------------------+
|               title|
+--------------------+
|            Zokkomon|
|   Zindagi Tere Naam|
|Zindagi Na Milegi...|
+--------------------+
only showing top 3 rows
```

The query returns all the movie titles that were released post-2010 and also sorts them in descending order by title. We then print three rows from the result. The `sql()` method returns the results as a DataFrame. Spark SQL offers an <a href="https://en.wikipedia.org/wiki/SQL:2003" target="_blank">ANSI:2003-compliant</a> SQL interface and the query we executed should look and feel similar to a query against a relational database table such as MySQL or PostgreSQL. We can also express the same query using DataFrames API as follows:


```shell
movies.where($"releaseYear" > 2010)
      .sort(desc("title"))
```

# Complex query example

Let’s try a slightly more complex query. Imagine that  we want to list all the movies released post 2010 and at the same time label each movie below average, average, and above average if the `hitFlop` column is less than 5, equal to 5, and higher than 5, respectively. The query and its output are presented below:

Work through an example that demonstrates executing queries using Spark SQL.

Spark Overview

DataFrames

Datasets

Spark SQL

Summary

Spark SQL - An Example

Simple query example