Gain insights into Spark Java API, learn about data transformations and SQL operations, and discover how to integrate big data and Java for scalable, high-speed processing.

tarball.tar.gz

DataFrameBasicsRun

SparkDataframeExampleDB

This course serves as a comprehensive introduction to the Spark Java API. Experienced Java developers will use object-oriented programming (OOP) principles to apply theory to Apache Spark and big data practice.

You’ll learn the basic components and architecture of Spark, a leading framework for building big data applications, before implementing them in Java. You’ll also explore data transformations like grouping, sorting, and joining. Further, you’ll learn to support SQL operations in the database and create a big data and batch template application with Java.

By the end of the course, you’ll be familiar with Apache Spark and know how to integrate big data with Java environments through the Spark Java API. You’ll wrap up by learning about monitoring and support functions for a live Spark Java environment.

Combining a leading big data framework with a leading programming language, this course will empower you to work efficiently with large volumes of data, and process at scale and speed.

Mastering Big Data with Apache Spark and Java

# Maven projects
For the small projects, and even for the main batch application template project, this course abides by a standard project structure based on Maven, and specifically relies on the pom.xml file for the configuration.

For this reason, this lesson focuses on describing the main Maven project structure parts, which can also serve as a guide in the future to come back to if we need a Maven-based Spark type of project.

The core structure of the Maven projects follows the usual Maven conventions centered around the so-called **Project Object Model (POM)** file, so it can be helpful to revise this first briefly.

[The previous project](https://www.educative.io/courses/mastering-big-data-apache-spark-java-api/running-the-first-spark-program) POM XML file can be a good starting point to understand the configurations explained in this lesson:




<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.jmb.examples</groupId>
    <artifactId>dataframe-basics</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <java.version>1.8</java.version>
        <scala.version>2.12</scala.version>
        <spark.version>3.0.0</spark.version>
        <maven-compiler-plugin.version>3.8.0</maven-compiler-plugin.version>
    </properties>

    <dependencies>
        
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_${scala.version}</artifactId>
            <version>${spark.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_${scala.version}</artifactId>
            <version>${spark.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>slf4j-simple</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.11</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>${maven-compiler-plugin.version}</version>
                <configuration>
                    <source>${java.version}</source>
                    <target>${java.version}</target>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.codehaus.mojo</groupId>
                <artifactId>exec-maven-plugin</artifactId>
                <version>1.6.0</version>
                <configuration>
                    <executable>java</executable>
                    <arguments>
                        <argument>-classpath</argument>
                        <classpath />
                        <argument>com.jmb.DataFrameBasicsMain</argument>
                    </arguments>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-assembly-plugin</artifactId>
                <version>2.4.1</version>
                <configuration>

                    <descriptorRefs>
                        <descriptorRef>jar-with-dependencies</descriptorRef>
                    </descriptorRefs>

                    <archive>
                        <manifest>
                            <mainClass>com.jmb.DataFrameBasicsMain</mainClass>
                        </manifest>
                    </archive>

                </configuration>
                <executions>
                    <execution>
                        <id>make-assembly</id>

                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

# Basics of POM

The POM is perhaps the most fundamental unit of work in Maven. It is an XML file that contains information about the project and configuration details that Maven utilises while building a project.

“Building” is understood in this context as a set of tasks, or goals, that eventually can produce either a packaged Java executable application, a Java library, or both. However, a single JAR file will not be the outcome of the Maven execution process.

> **Note:** For more information about Maven, the  [official documentation](https://maven.apache.org/guides/getting-started/index.html) can serve as an excellent manual to consult when needed.

Besides the configurations that can be manually set up in the POM XML file, Maven also works behind the scenes by following certain default conventions. These are also configurable and can be overwritten, but only if they are specified within this file.

One example of these is the build directory, the directory where the built object is copied to, denoted by the `<target>` XML tag. Another one is the source directory, the directory where the .java files reside, defaulted to the `src/main/java` path in the project’s folder structure unless changed explicitly.

When a Maven goal, like `package` or `test`, is executed, the POM XML file is analyzed and used to retrieve the needed configurations. This is part of several other tasks that occur without the developer's intervention, and are part of regular Maven execution.

## Minimal POM

The minimal POM contains the basic necessary configurations to describe a project that Maven can “understand.” These are the following:

```
<project>
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.jmb.examples</groupId>
    <artifactId>dataframe-basics</artifactId>
    <version>1.0-SNAPSHOT</version>
</project>
```

Leaving aside the `<modelVersion>` XML tag, the properties shown represent the following:
- The ID of the project’s group (many projects can be part of a group of related projects, as logical separations).
- The ID of the artifact, meaning, the project in Maven terms, but also the final packaged application.
- The version of the artifact under the specified group.


The values of all these three properties combined form the fully qualified artifact name, which in Maven jargon refers to a composite way of identifying executables and libraries uniquely. This is also helpful if we want to search for other libraries or programs in a Maven central repository.

## Projects' properties

The properties part of the pom.xml file lists reusable values within this same file. These values usually refer to the versions of dependencies (linked libraries) and plugins (Maven’s pluggable components containing additional functionality) used in a project. The following is an example of this taken from the same project we used before:

```
<properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <java.version>1.8</java.version>
        <scala.version>2.12</scala.version>
        <spark.version>3.0.0</spark.version>
        <maven-compiler-plugin.version>3.8.0</maven-compiler-plugin.version>
    </properties>
```

Here are some remarks about this section of the POM file regarding this course’s projects:

- The Spark version (`spark.version` property) used throughout this course is the 3.0.0 version.
- We need the Scala version because Spark is built on Scala and must be compatible with the specific Spark version (3.0.0).

- The Maven compiler plugin and Java version are defaulted to the values shown above.

One thing worth noting is that the properties’ names are defined by the developer, and are not standard to Maven. We use their values elsewhere in the POM XML file by abiding by the following convention:

```
<dependencies>
        
        <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_${scala.version}</artifactId>
        <version>${spark.version}</version>
        </dependency>

//POM file continues...
```

<br/>

# Spark dependencies

Maven dependencies are other files, projects, or libraries (such as jar, zip, etc.) that a project might need to compile, build, test, and run.

>**Note:** The Maven dependencies topic is, of course, of greater complexity, and the following [official dependencies docs](https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html) can serve as an introduction.


There are two main dependencies we use for the duration of this course: the Spark-core dependency, which contains the core component functionality and well as the older parts of Spark, and the Spark-SQL dependency, which includes the bulk of the DataFrame/Dataset functionality.

```

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_${scala.version}</artifactId>
            <version>${spark.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_${scala.version}</artifactId>
            <version>${spark.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>slf4j-simple</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
```

Here are a couple of points worth mentioning:

* The Spark version is used as an overall Spark version on the `<version>` tag. In contrast, the components or modules version uses the current compatible version of Scala for this Spark version. This can work as a good reminder of which related dependencies we can mix and match if needed.

* The `slf4j-simple` library is excluded (see the `<exclusions>` tags) because Spark ships with its own logging dependency to avoid compatibility issues.

# Maven plugins

Besides the usual Maven plugins to build and package the application into a single executable jar (see `maven-assembly-plugin`), the project includes an `exec-maven-plugin` plugin. This delegates the program execution to Maven and takes only one argument, which is the class with the main method we want to execute:


<br/>

``` 
            <plugin>
                <groupId>org.codehaus.mojo</groupId>
                <artifactId>exec-maven-plugin</artifactId>
                <version>1.6.0</version>
                <configuration>
                    <executable>java</executable>
                    <arguments>
                        <argument>-classpath</argument>
                        <classpath />
                        <argument>com.jmb.DataFrameBasicsMain</argument>
                    </arguments>
                </configuration>
            </plugin>
```

In the above code snippet, the main class is the one used in the previous lesson, `DataFrameBasicsMain.java`. Subsequent projects have a different class because whichever we choose is the entry point of a particular Spark application.

The course widgets use the plugin in the background, which we can trigger by running the following Maven command:

```
mvn [install] exec:exec
```

The `install` goal is usually recommended to trigger a pre-execution Project build. 

> **Note:** The Spark version used in this course (3.0.0) runs with the Java 1.8 version. For compatibility reasons, we should install and configure this Java version together with Maven. The alternative is to use other Java/Maven versions mutually supported by Spark 3.0.0.

Concise, straightforward, and powerful, the Spark Maven projects used in this course do not require any other plugins or essential dependencies to be built, run, or packaged into a jar that ships as a self-contained Spark executable application.

Naturally, the application can be enriched as much as needed if it grows in complexity, which tends to be the case in most applications’ development lifecycles.

Nevertheless, the Maven configurations described in the projects’ pom file so far are a solid starting point for getting a Spark-based project up and running.

Get introduced to the common structure of the Maven projects we'll use in the code examples and projects.

Course Introduction

Spark Introduction and Basics

Getting Started with Spark

DataFrame Basic Operations

DataFrame Advanced Operations

Spark SQL and Other Functionalities

Building a Big Data Batch Application

Deployment and Cluster Execution

Monitoring and Performance Fundamentals

Conclusion

Apendix

Spark Maven Based Projects

Maven projects