tarball.tar.gz

DataFrameBasicsRun

SparkDataframeExampleDB

This course serves as a comprehensive introduction to the Spark Java API. Experienced Java developers will use object-oriented programming (OOP) principles to apply theory to Apache Spark and big data practice.

You’ll learn the basic components and architecture of Spark, a leading framework for building big data applications, before implementing them in Java. You’ll also explore data transformations like grouping, sorting, and joining. Further, you’ll learn to support SQL operations in the database and create a big data and batch template application with Java.

By the end of the course, you’ll be familiar with Apache Spark and know how to integrate big data with Java environments through the Spark Java API. You’ll wrap up by learning about monitoring and support functions for a live Spark Java environment.

Combining a leading big data framework with a leading programming language, this course will empower you to work efficiently with large volumes of data, and process at scale and speed.

Mastering Big Data with Apache Spark and Java

## Ingesting data from files in Spark

Ingesting data is the first stage of a Big Data pipeline, and in many cases, it determines the initial step in a typical batch process. Spark offers developers a wide range of options when it comes to ingesting files from different formats.


The formats studied in this lesson are exemplified with a single project.  Having a separate project for each of them would be overkill, and the operations are fairly concise.

Spark fulfills the loading functionality internally by using parsers, so let’s highlight the essential steps of using a parser:

1. The input of a parser is the path of a file. This path can also be a regular expression, which enables the developer to load multiple files at once.


2. Parsers take options as extra arguments, which we showcase in our examples, but the values for these options are case sensitive, so “myPath” and “mypath” are not considered the same.


The below widget contains the project for this lesson:


# Ingesting data from files in Spark

Ingesting data is the first stage of a Big Data pipeline, and in many cases, it determines the initial step in a typical batch process. Spark offers developers a wide range of options when it comes to ingesting files from different formats.


The formats studied in this lesson are exemplified with a single project.  Having a separate project for each of them would be overkill, and the operations are fairly concise.

Spark fulfills the loading functionality internally by using parsers, so let’s highlight the essential steps of using a parser:

1. The input of a parser is the path of a file. This path can also be a regular expression, which enables the developer to load multiple files at once.


2. Parsers take options as extra arguments, which we showcase in our examples, but the values for these options are case sensitive, so “myPath” and “mypath” are not considered the same.


The below widget contains the project for this lesson:


Learn how to ingest information from files, in several formats, using Spark.

Course Introduction

Spark Introduction and Basics

Getting Started with Spark

DataFrame Basic Operations

DataFrame Advanced Operations

Spark SQL and Other Functionalities

Building a Big Data Batch Application

Deployment and Cluster Execution

Monitoring and Performance Fundamentals

Conclusion

Apendix

How to Ingest Files: Part I

Ingesting data from files in Spark