Gain insights into Spark Java API, learn about data transformations and SQL operations, and discover how to integrate big data and Java for scalable, high-speed processing.

tarball.tar.gz

DataFrameBasicsRun

SparkDataframeExampleDB

This course serves as a comprehensive introduction to the Spark Java API. Experienced Java developers will use object-oriented programming (OOP) principles to apply theory to Apache Spark and big data practice.

You’ll learn the basic components and architecture of Spark, a leading framework for building big data applications, before implementing them in Java. You’ll also explore data transformations like grouping, sorting, and joining. Further, you’ll learn to support SQL operations in the database and create a big data and batch template application with Java.

By the end of the course, you’ll be familiar with Apache Spark and know how to integrate big data with Java environments through the Spark Java API. You’ll wrap up by learning about monitoring and support functions for a live Spark Java environment.

Combining a leading big data framework with a leading programming language, this course will empower you to work efficiently with large volumes of data, and process at scale and speed.

Mastering Big Data with Apache Spark and Java

# Spark and databases

Though there are many different DataSources involved in big data processing, relational databases can still be the de facto choice as a data repository, specifically in situations where the business domains require data normalization, relationships between domain models, strongly consistent transactions, and so on.

Spark offers the possibility of interacting with RDBMS to load a whole table, filter information, and load a fraction of a table by executing queries on the table. It also provides functionality to run operations while ingesting from the database, such as filtering and aggregation, that help minimize the volumes of data retrieved.

In regards to minimizing the amount of data retrieved, one sensible strategy is to filter information at the database level while querying tables, if possible. The immediate benefit of this strategy is the reduction of data volumes transferred from the source.


> **Note:** We should always think in big data terms. We should assume that volumes of information are always huge, and aim at efficiency in every possible corner of our application.

Java applications always use a dialect while interacting with a database, and Spark incorporates the same principle. In fact, Spark establishes a connection to a database by using Java’s JDBC drivers for a particular vendor (such as PostgreSQL or MySQL)


For this reason, the JDBC drivers for a particular database vendor (or a dialect) have to be available in the worker nodes of the cluster or locally in the classpath, if doing standalone developments.

The following illustration summarizes Spark interaction with the database:



Let’s delve into loading information from a database.


Ingesting Databases

Let’s delve into loading information from a database.

Course Introduction

Spark Introduction and Basics

Getting Started with Spark

DataFrame Basic Operations

DataFrame Advanced Operations

Spark SQL and Other Functionalities

Building a Big Data Batch Application

Deployment and Cluster Execution

Monitoring and Performance Fundamentals

Conclusion

Apendix

Ingesting Databases

Spark and databases