tarball.tar.gz

DataFrameBasicsRun

SparkDataframeExampleDB

This course serves as a comprehensive introduction to the Spark Java API. Experienced Java developers will use object-oriented programming (OOP) principles to apply theory to Apache Spark and big data practice.

You’ll learn the basic components and architecture of Spark, a leading framework for building big data applications, before implementing them in Java. You’ll also explore data transformations like grouping, sorting, and joining. Further, you’ll learn to support SQL operations in the database and create a big data and batch template application with Java.

By the end of the course, you’ll be familiar with Apache Spark and know how to integrate big data with Java environments through the Spark Java API. You’ll wrap up by learning about monitoring and support functions for a live Spark Java environment.

Combining a leading big data framework with a leading programming language, this course will empower you to work efficiently with large volumes of data, and process at scale and speed.

Mastering Big Data with Apache Spark and Java

## Monitoring in SparkUI

The SparkUI is a user interface (UI) provided by the Spark libraries that allows the developer to query and inspect both the status of jobs and resources usage.

> **Note:** By default found on port `4040`, though configurable, the tool’s address is expressed as URL:PORT, where URL is the master node or cluster manager IP address.


When used and understood correctly, SparkUI is a powerful tool to detect bottlenecks in a Spark application and draw conclusions as to how resources are utilized (or underutilized.)

One advantage over reading the logs is that the information is displayed in real-time; that is, the different visualizations present in the tool allow the developer to monitor a running Spark application as well as querying finished applications’ historical information.

Nevertheless, a combination of log scrutiny and conclusions drawn from observation of the SparkUI screens can always provide a more comprehensive picture.

Let’s go through the different parts of the UI application by running one of the earliest projects used in this course, albeit with some code removed to make things simpler:


# Monitoring in SparkUI

The SparkUI is a user interface (UI) provided by the Spark libraries that allows the developer to query and inspect both the status of jobs and resources usage.

> **Note:** By default found on port `4040`, though configurable, the tool’s address is expressed as URL:PORT, where URL is the master node or cluster manager IP address.


When used and understood correctly, SparkUI is a powerful tool to detect bottlenecks in a Spark application and draw conclusions as to how resources are utilized (or underutilized.)

One advantage over reading the logs is that the information is displayed in real-time; that is, the different visualizations present in the tool allow the developer to monitor a running Spark application as well as querying finished applications’ historical information.

Nevertheless, a combination of log scrutiny and conclusions drawn from observation of the SparkUI screens can always provide a more comprehensive picture.

Let’s go through the different parts of the UI application by running one of the earliest projects used in this course, albeit with some code removed to make things simpler:


Get introduced to a useful tool included and shipped with the Spark library: SparkUI.

Course Introduction

Spark Introduction and Basics

Getting Started with Spark

DataFrame Basic Operations

DataFrame Advanced Operations

Spark SQL and Other Functionalities

Building a Big Data Batch Application

Deployment and Cluster Execution

Monitoring and Performance Fundamentals

Conclusion

Apendix

Cluster Monitoring and SparkUI

Monitoring in SparkUI