Home>Courses>Mastering Big Data with Apache Spark and Java

Mastering Big Data with Apache Spark and Java

Gain insights into Spark Java API, learn about data transformations and SQL operations, and discover how to integrate big data and Java for scalable, high-speed processing.

Intermediate

55 Lessons

17h 15min

Certificate of Completion

Gain insights into Spark Java API, learn about data transformations and SQL operations, and discover how to integrate big data and Java for scalable, high-speed processing.

AI-POWERED

Explanations

AI-POWERED

Explanations

This course includes

30 Playgrounds

7 Quizzes

Course Overview

What You'll Learn

Course Content

Apply Your Skills

Course Overview

This course serves as a comprehensive introduction to the Spark Java API. Experienced Java developers will use object-oriented programming (OOP) principles to apply theory to Apache Spark and big data practice. You’ll learn the basic components and architecture of Spark, a leading framework for building big data applications, before implementing them in Java. You’ll also explore data transformations like grouping, sorting, and joining. Further, you’ll learn to support SQL operations in the database and cre...Show More

What You'll Learn

Learn Apache Spark fundamentals and gain an overview of its building blocks

Learn Advanced Transformations and leverage Spark’s powerful library, Spark SQL

Acquire practical experience through examples, coding, and recipes

Develop a big data batch application with foundations in both design patterns and good programming practices using Spark

Learn Apache Spark fundamentals and gain an overview of its building blocks

Course Content

Course Introduction

2 Lessons

Get familiar with Apache Spark, its architecture, Java API, and big data processing.

About the Course

Course Structure

Spark Introduction and Basics

5 Lessons

Get started with Apache Spark's architecture, in-memory computing, and scalable data processing.

Spark Fundamentals

Components and Architecture

Spark and Big Data

Spark's Java Main Abstraction: The DataFrame

Quiz Yourself: Spark Introduction

Getting Started with Spark

5 Lessons

Explore setting up and running Spark programs, configuring Maven projects, and utilizing DataFrames.

Running the First Spark Program

Spark Maven Based Projects

Enriching the Basic DataFrame Program

Deep Dive: Transformations and Data Storage

Quiz Yourself: Getting Started with Spark

DataFrame Basic Operations

9 Lessons

Break down complex ideas for effective manipulation of DataFrames and Datasets in Spark.

Working with DataFrame's Schemas

Dataset: a DataFrame of POJOs

Transformations and Actions

Transformations (I): Map and Filter

Actions (I): Count, Take, and Collect

Deep Dive: Internals of Spark Execution

Transformations (II): FlatMap and Distinct

Actions (II): Reduce and Aggregate Functions: Max, Min, and Mean

Quiz Yourself: Schema of a DataFrame

DataFrame Advanced Operations

8 Lessons

Deepen your knowledge of advanced DataFrame operations like partitioning, joins, and UDFs in Spark.

Data Partitioning and Shuffling

The groupBy and groupByKey methods

Joins

Sort and OrderBy

Union, UnionByName, and DropDuplicates

Accumulators and Broadcast Variables

UDFs: User-defined Functions

Quiz Yourself: DataFrame Advanced Operations

Spark SQL and Other Functionalities

8 Lessons

Follow the process of leveraging Spark's SQL, schema manipulation, file/database ingestion, and serialization.

Spark SQL Goodness

Schema Manipulation

How to Ingest Files: Part I

How to Ingest Files: Part II

Ingesting Databases

Exporting Information

Serialization: Working through the Wire

Quiz Yourself: SparkSQL and Other Functionalities

Building a Big Data Batch Application

8 Lessons

Piece together the parts of building a Spark batch application, including architecture, driver program design, ingestion, and testing.

The Application Architecture Ecosystem

Driver Program Design and Project Structure

Driver Program and Job Implementation: Part I

Driver Program and Job Implementation: Part II

Ingestion Job: Part I

Ingestion Job: Part II

Batch Pipelines and Other Types of Jobs

Testing and Spark

Deployment and Cluster Execution

3 Lessons

Try out executing and deploying Apache Spark applications in local and cluster modes.

Local and Cluster-based Execution

Deploying and Running a Spark Application

Quiz Yourself: Deployment and Cluster Execution

Monitoring and Performance Fundamentals

4 Lessons

Unpack the core of interpreting Spark logs, using SparkUI, and fundamental performance optimization techniques.

Interpreting Spark Logs

Cluster Monitoring and SparkUI

Performance Fundamentals and Recipes

Quiz Yourself: Monitoring and Performance Fundamentals

10.

Conclusion

1 Lessons

Examine further resources for Spark and Java development to continue your learning.

Where to Go from Here

11.

Apendix

2 Lessons

Break down complex tools and techniques for local Spark development and debugging using IntelliJ.

Tools to Develop in Spark Locally

IntelliJ: Debug and Inspect Spark Execution

Course Author

Juan Bruno

Project

Apriori Algorithm for Finding Frequent Itemsets with PySpark

intermediate

1 hour

Project

Apriori Algorithm for Finding Frequent Itemsets with PySpark

Learn to apply the Apriori algorithm using PySpark for efficient analysis of frequent itemsets in retail data.

intermediate

1 hour

Trusted by 2.5 million developers working at companies

"These are high-quality courses. Trust me. I own around 10 and the price is worth it for the content quality. EducativeInc came at the right time in my career. I'm understanding topics better than with any book or online video tutorial I've done. Truly made for developers. Thanks"

Anthony Walker

@_webarchitect_

"Just finished my first full #ML course: Machine learning for Software Engineers from Educative, Inc. ... Highly recommend!"

Evan Dunbar

ML Engineer

"You guys are the gold standard of crash-courses... Narrow enough that it doesn't need years of study or a full blown book to get the gist, but broad enough that an afternoon of Googling doesn't cut it."

Carlos Matias La Borde

Software Developer

"I spend my days and nights on Educative. It is indispensable. It is such a unique and reader-friendly site"

Souvik Kundu

Front-end Developer

"Your courses are simply awesome, the depth they go into and the breadth of coverage is so good that I don't have to refer to 10 different websites looking for interview topics and content."

Vinay Krishnaiah

Software Developer

"I've tried probably 5-7 different sites and Educative is easily the best. It perfectly blends explanation with interactivity"

Eric Downs

Musician/Entrepeneur

Hands-on Learning Powered by AI

See how Educative uses AI to make your learning more immersive than ever before.

Instant Code Feedback

Evaluate and debug your code with the click of a button. Get real-time feedback on test cases, including time and space complexity of your solutions.

AI-Powered Mock Interviews

Put your skills to the test in a simulated interview setting. Receive personalized feedback based on your performance. Available in Premium & Premium Plus plans.

Adaptive Learning

At various checkpoints throughout Educative courses, you will be prompted to take a quick assessment. Receive a condensed curriculum tailored to your strengths and skill gaps.

Explain with AI

Select any text within any Educative course, and get an instant explanation — without ever leaving your browser.

AI Code Mentor

AI Code Mentor helps you quickly identify errors in your code, learn from your mistakes, and nudge you in the right direction — just like a 1:1 tutor!

Free Resources

FOR TEAMS

Interested in this course for your business or team?

Unlock this course (and 1,000+ more) for your entire org with DevPath