Explore

EXPLORE THE CATALOGSupercharge your career with 700+ hands-on courses

View All Courses

Python

Java

JavaScript

C

React

Docker

Vue JS

R

Web Dev

DevOps

AWS

C#

LEARNING TOOLSExplore the industry's most complete learning platform

CoursesLevel up your skills

Cloud LabsSetup-free practice with Cloud Services

Skill PathsAchieve learning goals

ProjectsBuild real-world applications

AssessmentsBenchmark your skills

Personalized PathsGet the right resources for your goals

LEARN TO CODE

Check out our beginner friendly courses.

Pricing

For Business

Resources

BlogFor developers, By developers

CheatsheetsDownload handy guides for tech topics

WebinarsSessions with our global developer community

AnswersTrusted answers to developer questions

Search⌘ K

Join for free

Home>Courses>Building Scalable Data Pipelines with Kafka

Building Scalable Data Pipelines with Kafka

This course brings any Big Data newbie up to speed on one of the most widely used open-source platform to build data pipelines - Apache Kafka.

Beginner

62 Lessons

3h

Certificate of Completion

This course brings any Big Data newbie up to speed on one of the most widely used open-source platform to build data pipelines - Apache Kafka.

AI-POWERED

Explanations

AI-POWERED

Explanations

This course includes

8 Playgrounds

Course Overview

What You'll Learn

Course Content

Apply Your Skills

Course Overview

If you’re interested in Big Data, then Apache Kafka is a must-know tool. What started as an internal LinkedIn project to streamline data transmission and propagation among services has quickly grown to become a mainstay platform for building highly scalable data pipelines. Meet Apache Kafka - the ubiquitous tool to build pipelines for diverse use cases ranging from chronologically tracking user-activity on a website to implementing publish-subscribe feeds. This course introduces you to Kafka theory and ...Show More

If you’re interested in Big Data, then Apache Kafka is a must-know tool. What started as an internal LinkedIn project to streamline data transmission and propagation among services has quickly grown to become a mainstay platform for building highly scala...Show More

What You'll Learn

Learn the theory behind Kafka

Interact with a Kafka cluster running in the browser-terminal

Learn the theory behind Kafka

Show more

Course Content

1.

Basics

7 Lessons

Step through the fundamentals of Kafka, distributed systems, messaging patterns, and core components.

Characteristics of Distributed Systems

Messaging Patterns

Components of Kafka

Partition Rebalancer

2.

Kafka Producer

4 Lessons

Unpack the core of Kafka Producers, message sending methods, configurations, and serialization techniques.

Sending Messages

Producer Configurations

Producer Serialization

3.

Kafka Consumer

9 Lessons

Go hands-on with Kafka consumers, configurations, offsets, and partition rebalancing techniques.

Consumer and Consumer Groups

Consumer Configurations

Commits and Offsets

Offset Commit Configuration

Handling Rebalances

Stopping a Consumer

Consumer Deserializer

Running a Single Consumer

4.

Kafka Internals

7 Lessons

Break down complex ideas of Kafka's replication, controller, request processing, and reliability.

Kafka Controller

Request Processing

Partition Allocation

Reliability in Kafka

Reliable Producers and Consumers

5.

Conclusion

1 Lessons

Compare Kafka's scalability, throughput, and real-time processing with other messaging systems.

Kafka vs Other Messaging Systems

6.

Appendix

3 Lessons

Activate Zookeeper insights, practical API use, and common distributed system solutions.

Zookeeper: Intro

Zookeeper: Example

Zookeeper: Practical

7.

Reference: Replication

14 Lessons

Master the principles of replica management, leader-based and leaderless replication strategies, and conflict resolution methods.

Single Leader Replication

Asynchronous VS Synchronous Replication

Followers in Log Replication

Log Replication

Issues in Single Leader Replication

Multi Leader Replication

Issues in Multi Leader Replication

Multi-Leader Topologies

Leaderless Replication

Quorum Variations

Concurrent Writes

Video Streaming Queue a Concurrent Writes Example

8.

Reference: Partitioning

4 Lessons

Learn how to use partitioning strategies to enhance scalability and optimize data pipelines.

Partitioning Schemes

Secondary Indexes

Number of Partitions

9.

Reference: Transactions

9 Lessons

Discover the logic behind managing data transactions, isolation levels, and concurrent write challenges.

Introduction to Transactions

More on Transactions

Isolation Levels

Read Skew and Snapshot Isolation

Concurrent Writes and Lost Updates

Serializability

Two Phase Locking

Serializable Snapshot Isolation

10.

Reference: Issues in Distributed Systems

4 Lessons

Examine the challenges in developing and maintaining distributed systems, including networking, time synchronization, and handling failures.

Agreeing On Time

Working with Time Issues

Course Author

Real-Time Analysis of a Simulated E-commerce Data Stream

Project

Real-Time Analysis of a Simulated E-commerce Data Stream

Real-Time Analysis of a Simulated E-commerce Data Stream

Learn to design a real-time data pipeline for e-commerce using modern tools like Kafka and Grafana on AWS.

intermediate

Trusted by 2.5 million developers working at companies

"These are high-quality courses. Trust me. I own around 10 and the price is worth it for the content quality. EducativeInc came at the right time in my career. I'm understanding topics better than with any book or online video tutorial I've done. Truly made for developers. Thanks"

Anthony Walker

@_webarchitect_

"Just finished my first full #ML course: Machine learning for Software Engineers from Educative, Inc. ... Highly recommend!"

Evan Dunbar

ML Engineer

"You guys are the gold standard of crash-courses... Narrow enough that it doesn't need years of study or a full blown book to get the gist, but broad enough that an afternoon of Googling doesn't cut it."

Carlos Matias La Borde

Software Developer

"I spend my days and nights on Educative. It is indispensable. It is such a unique and reader-friendly site"

Souvik Kundu

Front-end Developer

"Your courses are simply awesome, the depth they go into and the breadth of coverage is so good that I don't have to refer to 10 different websites looking for interview topics and content."

Vinay Krishnaiah

Software Developer

"I've tried probably 5-7 different sites and Educative is easily the best. It perfectly blends explanation with interactivity"

Eric Downs

Musician/Entrepeneur

Hands-on Learning Powered by AI

See how Educative uses AI to make your learning more immersive than ever before.

Instant Code Feedback

Evaluate and debug your code with the click of a button. Get real-time feedback on test cases, including time and space complexity of your solutions.

AI-Powered Mock Interviews

Put your skills to the test in a simulated interview setting. Receive personalized feedback based on your performance. Available in Premium & Premium Plus plans.

Adaptive Learning

At various checkpoints throughout Educative courses, you will be prompted to take a quick assessment. Receive a condensed curriculum tailored to your strengths and skill gaps.

Explain with AI

Select any text within any Educative course, and get an instant explanation — without ever leaving your browser.

AI Code Mentor

AI Code Mentor helps you quickly identify errors in your code, learn from your mistakes, and nudge you in the right direction — just like a 1:1 tutor!

Free Resources

FOR TEAMS

Interested in this course for your business or team?

Unlock this course (and 1,000+ more) for your entire org with DevPath

Learn in-demand tech skills in half the time

Copyright ©2024 Educative, Inc. All rights reserved.

soc2