Gain insights into Spark Java API, learn about data transformations and SQL operations, and discover how to integrate big data and Java for scalable, high-speed processing.

tarball.tar.gz

DataFrameBasicsRun

SparkDataframeExampleDB

This course serves as a comprehensive introduction to the Spark Java API. Experienced Java developers will use object-oriented programming (OOP) principles to apply theory to Apache Spark and big data practice.

You’ll learn the basic components and architecture of Spark, a leading framework for building big data applications, before implementing them in Java. You’ll also explore data transformations like grouping, sorting, and joining. Further, you’ll learn to support SQL operations in the database and create a big data and batch template application with Java.

By the end of the course, you’ll be familiar with Apache Spark and know how to integrate big data with Java environments through the Spark Java API. You’ll wrap up by learning about monitoring and support functions for a live Spark Java environment.

Combining a leading big data framework with a leading programming language, this course will empower you to work efficiently with large volumes of data, and process at scale and speed.

Mastering Big Data with Apache Spark and Java

# Working locally

One advantageous property of using Spark as a Java library is that the code can be run and, more interestingly, debugged in real-time.

Debugging in particular is very valuable because it allows developers to conduct local live-testing, and to understand, through inspection of Spark objects, the nature of Spark-based applications.

This lesson demonstrates how to debug a Spark-based project, so let’s start right away.


>**Note:** The lesson illustrates the necessary actions in IntelliJ IDEA IDE. Feel free to investigate plenty of options on the Web for debugging in Eclipse or NetBeans, if that happens to be the preferred “flavor” for a development environment.




# Debugging the basic example

Let’s briefly recall the lesson “[Running The First Spark Program](https://www.educative.io/courses/mastering-big-data-apache-spark-java-api/running-the-first-spark-program)."

To load the project from the code widget into IntelliJ IDEA IDE locally, the first step involves creating a simple Maven-based project, as described in the following [JetBrains Documentation](https://www.jetbrains.com/help/idea/maven-support.html).

Then, the necessary dependencies from the widget’s `pom.xml` file can be copied into the local project file with the same name. This allows Maven to locate and download the necessary Spark artifacts (libraries). These dependencies are always listed within the following XML tag:



```
<dependencies>
//Dependencies listed within this XML element...
</dependencies>
```
Lastly, the project’s folder structure—that is, the way the files are organized into specific folders following a Maven convention for basic Java projects—has to be the same, so then the necessarily related classes can be found in the same Java packages defined in the code widget when execution takes place.
 

More information on this can be found at this [Maven Docs](https://maven.apache.org/guides/introduction/introduction-to-the-standard-directory-layout.html) link.

>**Note:** If the folder structure of the project is changed, the codebase has to be reorganized, including the “Resources” folder containing the file(s) to be ingested by Spark.




## Debugging projects

We can proceed to debug the main class of the application, named `DataFrameBasicsMain`, by placing first a breakpoint on the following code area:


 

Learn how to set up and debug the projects on an IntelliJ IDEA Integrated Development Environment (IDE).

Course Introduction

Spark Introduction and Basics

Getting Started with Spark

DataFrame Basic Operations

DataFrame Advanced Operations

Spark SQL and Other Functionalities

Building a Big Data Batch Application

Deployment and Cluster Execution

Monitoring and Performance Fundamentals

Conclusion

Apendix

IntelliJ: Debug and Inspect Spark Execution

Working locally