Gain insights into Spark, its architecture, application lifecycle, and APIs. Delve into data frames, datasets, and Spark SQL to effectively manage and query big data.

spark.tar.gz

SparkShellUI

SparkHistoryServerUI

Spark has come to dominate the big data processing space in a short span of time since its release and now serves as the de-facto unified big data processing engine in the industry. 

In this course, you will get a complete introduction to the basics of Spark. You will start by learning about the architecture, the application lifecycle, and its API.

From there, you will dive into the data frame data structure and its API as well as the strongly-typed datasets API. Lastly, you’ll get into the Spark SQL engine which will allow you to issue queries on structured data with a schema.

By the end of this course, you will have the confidence to use Spark in any of your big data projects.

An Introduction to Spark

So far, we have seen simple examples of user and built in functions, but Spark also offers functions that can work on more complex types. Let’s work through an example that’ll require us to use some of the higher-order functions. Realize that, given the structure of our data, we can’t answer questions by filtering on the actor’s name, since the actors in a movie appear as a pipe delimited string in the `actors` column. Instead, we’ll first need to split the string by the delimiter to get an array of string tokens where each token is an actor’s name starring in the movie. We can use the `split()` function as follows:

```shell
spark-sql> SELECT title, releaseYear, hitFlop, split(actors,"[|]") AS actors FROM movies;
```

For demonstrating the various tools that come with Spark, we have used the `spark-sql` CLI utility instead of `spark-shell` for this example. A few rows from the above query are shown below:




Albela	2001	2	["Govinda "," Aishwarya Rai Bachchan "," Jackie Shroff "," Namrata Shirodkar"]
Lagaan: Once Upon a Time in India	2001	6	["Aamir Khan "," Gracy Singh "," Rachel Shelley "," Paul Blackthorne"]
Meri Biwi Ka Jawab Nahin	2004	1	["Akshay Kumar "," Sridevi "," Gulshan Grover "," Laxmikant Berde"]

Note that the actors appear as an array instead of a pipe-delimited string. The actor names also have trailing spaces that we’ll fix later. We’ll save the result from the above query as a table, though we can also write nested queries, which may be harder to comprehend.

```shell
spark-sql> CREATE TABLE tempTbl1 AS SELECT title, releaseYear, hitFlop, split(actors,"[|]") AS actors FROM movies;
```

To answer questions based on actors, we'll need to access movie rows for each actor. Consider the movie `Albela` which has the actors listed as `["Govinda "," Aishwarya Rai Bachchan "," Jackie Shroff "," Namrata Shirodkar"]`. We need to duplicate this row for each actor. That is, we get four rows with the same columns except there’s a single actor name in the `actors` column. We can achieve this using the `explode()` function, which works on arrays and maps.

```shell
spark-sql> SELECT  title, releaseYear, hitFlop, explode(actors) AS actor FROM tempTbl1 LIMIT 10;
Albela	2001	2	Govinda
Albela	2001	2	 Aishwarya Rai Bachchan
Albela	2001	2	 Jackie Shroff
Albela	2001	2	 Namrata Shirodkar
Lagaan: Once Upon a Time in India	2001	6	Aamir Khan
Lagaan: Once Upon a Time in India	2001	6	 Gracy Singh
Lagaan: Once Upon a Time in India	2001	6	 Rachel Shelley
Lagaan: Once Upon a Time in India	2001	6	 Paul Blackthorne
Meri Biwi Ka Jawab Nahin	2004	1	Akshay Kumar
Meri Biwi Ka Jawab Nahin	2004	1	 Sridevi
Time taken: 0.104 seconds, Fetched 10 row(s)
```

We can save the above result in a new table `tempTbl2` and also apply the `trim()` function on the actors' names.

```shell
spark-sql> CREATE TABLE tempTbl2 AS SELECT title, releaseYear, hitFlop, trim(actor) AS actor FROM (SELECT  title, releaseYear, hitFlop, explode(actors) AS actor FROM tempTbl1);
```

A few rows from the `tempTbl2` are shown below:

```shell
spark-sql> select * from tempTbl2 limit 10;
Albela	2001	2	Govinda
Albela	2001	2	Aishwarya Rai Bachchan
Albela	2001	2	Jackie Shroff
Albela	2001	2	Namrata Shirodkar
Lagaan: Once Upon a Time in India	2001	6	Aamir Khan
Lagaan: Once Upon a Time in India	2001	6	Gracy Singh
Lagaan: Once Upon a Time in India	2001	6	Rachel Shelley
Lagaan: Once Upon a Time in India	2001	6	Paul Blackthorne
Meri Biwi Ka Jawab Nahin	2004	1	Akshay Kumar
Meri Biwi Ka Jawab Nahin	2004	1	Sridevi
Time taken: 0.063 seconds, Fetched 10 row(s)
```

Now we can issue queries specific to each actor. For example, we can compute the average rating for the movies an actor has starred in. The query and its output are shown below. We print only the first five rows for brevity.

```shell
spark-sql> SELECT actor, avg(hitFlop) AS score from tempTbl2 GROUP BY actor ORDER BY score desc LIMIT 5;
Tabrett Bethell	9.0
Nikitin Dheer	8.0
Satyaraj	8.0
M.K. Raina	8.0
Kiran Juneja	7.0
Time taken: 1.093 seconds, Fetched 5 row(s)
```

There are other high order functions such as `get_json_object()`, `from_json()`, `to_json()`, `explode()`, and `selectExpr()` that can be used to manipulate complex types. Here are some more examples:

```shell
## Say we want to collect all the movie titles from tempTbl2 as a list.
## We can use collect_list() function
select collect_list(title) AS allTitles from tempTbl2;
```

The above query will also contain duplicates, since the table `tempTbl2` contains exploded values. We can use the function `array_distinct()` to remove the duplicates as follows:

```shell
spark-sql> select array_distinct(allTitles) FROM (select collect_list(title) AS allTitles from tempTbl2);
```

For some of the operations, we can use UDFs instead of the built-in functions. There are also other functions for arrays such as `array_intersect()`, `array_join()`, and `array_union()`. Similarly, there are functions that operate on maps.

Some functions allow us to pass in anonymous lambda functions as arguments  that operate on each element of the array. Recall that the table `tempTbl1` had `actors` as an array of strings. We can convert all the strings values contained in each array to uppercase using the `transform()` function shown below:

```shell
spark-sql> select transform(actors, actor -> upper(actor)) from tempTbl1 LIMIT 5;
["GOVINDA "," AISHWARYA RAI BACHCHAN "," JACKIE SHROFF "," NAMRATA SHIRODKAR"]
["AAMIR KHAN "," GRACY SINGH "," RACHEL SHELLEY "," PAUL BLACKTHORNE"]
["AKSHAY KUMAR "," SRIDEVI "," GULSHAN GROVER "," LAXMIKANT BERDE"]
["SHAH RUKH KHAN "," MADHURI DIXIT "," SALMAN KHAN "," ATUL AGNIHOTRI"]
["SHAH RUKH KHAN "," JUHI CHAWLA "," JACKIE SHROFF "," NIRMAL PANDEY"]
Time taken: 0.087 seconds, Fetched 5 row(s)
```

The `transform()` function applies the lambda argument to each element of the array and returns a new array with the result. Other such functions are  `filter()`, `exists()`, and `reduce()`.



>All the queries and commands used in this lesson are reproduced in the widget below for easy copy and pasting into the terminal.

Explore examples to explore more complex functions in Spark SQL such as transform, map, explode, and more.

Spark Overview

DataFrames

Datasets

Spark SQL

Summary

Higher-Order Function