Gain insights into Spark Java API, learn about data transformations and SQL operations, and discover how to integrate big data and Java for scalable, high-speed processing.

tarball.tar.gz

DataFrameBasicsRun

SparkDataframeExampleDB

This course serves as a comprehensive introduction to the Spark Java API. Experienced Java developers will use object-oriented programming (OOP) principles to apply theory to Apache Spark and big data practice.

You’ll learn the basic components and architecture of Spark, a leading framework for building big data applications, before implementing them in Java. You’ll also explore data transformations like grouping, sorting, and joining. Further, you’ll learn to support SQL operations in the database and create a big data and batch template application with Java.

By the end of the course, you’ll be familiar with Apache Spark and know how to integrate big data with Java environments through the Spark Java API. You’ll wrap up by learning about monitoring and support functions for a live Spark Java environment.

Combining a leading big data framework with a leading programming language, this course will empower you to work efficiently with large volumes of data, and process at scale and speed.

Mastering Big Data with Apache Spark and Java

# Spark execution modes

All the examples shown so far execute in **interactive mode**, via a server or single machine or, provided the examples are run locally, via a laptop and an IDE (this latter option with the help of the Maven execution plugin).

Spark offers different modes to execute applications, each with various benefits and use cases, so let’s explain them briefly:

- **Local Mode**: This is the preferred option for developers becayse everything runs on the same environment or in a single machine (a laptop, PC, isolated server or machine, and so on).

- **Cluster Mode**: This was introduced in the "[Deep Dive: Internals of Spark Execution](https://www.educative.io/courses/mastering-big-data-apache-spark-java-api/deep-dive-internals-of-spark-execution)" lesson and allows for applications to run in a cluster managed by a resource manager. The cluster can be an on-premise, cloud, or even a local cluster (based on a set of Docker images, for example).

Analyzing both of these execution modes provides insight into how Spark works at both complex and straightforward levels when it comes to executing an application; this is also a valuable piece of information that can help choose when to run either in scenarios such as development, production, and so on.


# Local mode

The most significant benefit of running in local mode is perhaps the possibility of running a whole Spark application, with all the components involved, in a single resource such as a laptop or PC. This facilitates onboarding new engineers into a project and provides a way of quickly and easily testing the totality of our code locally. Furthermore, it offers the developers debug capabilities. This can be learned in the [IntelliJ: Debug and Inspect Spark Execution](https://www.educative.io/courses/mastering-big-data-apache-spark-java-api/intellij-debug-and-inspect-spark-execution) lesson in the Appendix chapter.

Spark takes care of the heavy lifting and creates both driver and executor processes required to run a `Job` alongside all the needed communication mechanisms between them, all done seamlessly.

## Running code in local mode

At this point in the course we might be well acquainted with running our code locally, but what are the main components or classes that need to be used and configured? Let’s take a quick glimpse at a few lines of familiar code:


Get introduced to execution modes in a Spark application, both locally and distributedly.


Local and Cluster-based Execution

Get introduced to execution modes in a Spark application, both locally and distributedly.

Course Introduction

Spark Introduction and Basics

Getting Started with Spark

DataFrame Basic Operations

DataFrame Advanced Operations

Spark SQL and Other Functionalities

Building a Big Data Batch Application

Deployment and Cluster Execution

Monitoring and Performance Fundamentals

Conclusion

Apendix

Local and Cluster-based Execution

Spark execution modes