Home>Courses>Data Engineering Foundations in Python

Data Engineering Foundations in Python

Gain insights into data engineering foundations, explore data life cycle stages, and delve into creating data pipelines using Python, Kafka, PySpark, Airflow, and dbt.

Beginner

46 Lessons

Certificate of Completion

Gain insights into data engineering foundations, explore data life cycle stages, and delve into creating data pipelines using Python, Kafka, PySpark, Airflow, and dbt.

AI-POWERED

Explanations

AI-POWERED

Explanations

This course includes

1 Project

57 Playgrounds

7 Quizzes

Course Overview

What You'll Learn

Course Content

Recommendations

Course Overview

Data engineering is currently one of the most in-demand fields in data and technology. It intersects software engineering, DataOps, data architecture, data management, and security. Data engineers, such as analysts and data scientists, lay the foundation to serve data for consumers. In this course, you will learn the foundation of data engineering, covering different parts of the entire data life cycle: data warehouse, ingestion, transformation, orchestration, etc. You will also gain hands-on experience b...Show More

TAKEAWAY SKILLS

Python

Sql

What You'll Learn

An understanding of the data engineering life cycle

Familiarity with the cloud data warehouse and data modeling techniques

Hands-on experience with data engineering tools such as GCP, Airflow, Spark, and dbt

The ability to build data pipelines from scratch in Python

An understanding of the data engineering life cycle

Course Content

Getting Started

1 Lessons

Get familiar with essential data engineering skills, including cloud architectures and pipeline orchestration.

Introduction

Data Team Structure

3 Lessons

Get started with the essentials of data team roles and effective team structures.

Data Team Roles

Data Team Structure: Embedded or Centralized?

Quiz: Data Team Structure

Data Engineering Life Cycle

5 Lessons

Master the steps to implement and manage data engineering life cycle stages using Google Cloud.

Set Up the Google Cloud

Introduction to Data Engineering Life Cycle

Ingestion, Transformation, and Visualization

Storage and Infrastructure

Quiz: Data Engineering Life Cycle

Cloud Data Architecture

6 Lessons

Grasp the fundamentals of cloud data evolution, service models, architectures, and best practices for efficiency and security.

Evolution of Cloud Data Architecture

Cloud Service Models

Cloud Data Architectures: Lambda, Kappa, and Data Warehouse

Cloud Data Architectures: Data Lake and Data Mesh

Patterns of Good Cloud Data Architecture

Quiz: Cloud Data Architecture

Data Ingestion

9 Lessons

Take a closer look at methods and tools for efficient data ingestion using Python.

Batch Ingestion

Stream Ingestion

Push vs. Pull

Ingestion Methods—SFTP, API, and Object Storage

Ingestion Methods—CDC

Ingestion Methods—Streaming Platform

Ingest with pandas

Ingest with PySpark

Quiz: Data Ingestion

Data Modeling

8 Lessons

Focus on mastering data modeling and SQL for effective database and analytical operations.

Introduction to Data Modeling

Kimball’s Dimensional Modeling

Steps of Dimensional Modeling

DDL vs. DML

Query Life Cycle

Challenge: BigQuery SQL

Solution: BigQuery SQL

Quiz: Data Modeling

Data Orchestration

8 Lessons

Master data orchestration using tools like Airflow, Dagster, and dbt to enhance workflows.

What Is Data Orchestration?

Airflow Basics

Airflow DAG Design

Challenge: Build an FXRate Reporting Pipeline with Airflow

Solution: Build an FXRate Reporting Pipeline with Airflow

Orchestration Tool: Dagster

Orchestration Tool: dbt

Quiz: Data Orchestration

Data Quality

4 Lessons

Solve problems in measuring and maintaining data quality with schema validation and testing.

Data Quality Measurement

Data Schema: Avro and Protobuf

Manage Data Quality in dbt

Quiz: Data Quality

Build an End-to-End Data Pipeline for Formula 1 Analysis

Project

10.

Epilogue

1 Lessons

Look at the fundamentals of data engineering, future paths, and mastering essential tools.

The Way Forward

11.

Appendix

1 Lessons

Explore additional GCP resources and billing management for efficient cloud utilization.

Additional Resources

Course Author

Xiaoxu Gao

Trusted by 2.5 million developers working at companies

"These are high-quality courses. Trust me. I own around 10 and the price is worth it for the content quality. EducativeInc came at the right time in my career. I'm understanding topics better than with any book or online video tutorial I've done. Truly made for developers. Thanks"

Anthony Walker

@_webarchitect_

"Just finished my first full #ML course: Machine learning for Software Engineers from Educative, Inc. ... Highly recommend!"

Evan Dunbar

ML Engineer

"You guys are the gold standard of crash-courses... Narrow enough that it doesn't need years of study or a full blown book to get the gist, but broad enough that an afternoon of Googling doesn't cut it."

Carlos Matias La Borde

Software Developer

"I spend my days and nights on Educative. It is indispensable. It is such a unique and reader-friendly site"

Souvik Kundu

Front-end Developer

"Your courses are simply awesome, the depth they go into and the breadth of coverage is so good that I don't have to refer to 10 different websites looking for interview topics and content."

Vinay Krishnaiah

Software Developer

"I've tried probably 5-7 different sites and Educative is easily the best. It perfectly blends explanation with interactivity"

Eric Downs

Musician/Entrepeneur

Hands-on Learning Powered by AI

See how Educative uses AI to make your learning more immersive than ever before.

Instant Code Feedback

Evaluate and debug your code with the click of a button. Get real-time feedback on test cases, including time and space complexity of your solutions.

AI-Powered Mock Interviews

Put your skills to the test in a simulated interview setting. Receive personalized feedback based on your performance. Available in Premium & Premium Plus plans.

Adaptive Learning

At various checkpoints throughout Educative courses, you will be prompted to take a quick assessment. Receive a condensed curriculum tailored to your strengths and skill gaps.

Explain with AI

Select any text within any Educative course, and get an instant explanation — without ever leaving your browser.

AI Code Mentor

AI Code Mentor helps you quickly identify errors in your code, learn from your mistakes, and nudge you in the right direction — just like a 1:1 tutor!

Recommended before starting this course

Course

Introduction to Big Data and Hadoop

Delve into Big Data essentials, explore data types, and gain insights into Hadoop components like YARN, MapReduce, HDFS, and Spark. Discover foundations to excel in the growing Big Data field.

beginner

10 hours

Course

An Introduction to Spark

Gain insights into Spark, its architecture, application lifecycle, and APIs. Delve into data frames, datasets, and Spark SQL to effectively manage and query big data.

beginner

2 hours 50 mins

Course

From Pandas to PySpark DataFrame

Gain insights into enhancing Python data processing with PySpark. Delve into reading, transforming, aggregating data, and creating user-defined functions, boosting efficiency with Apache Spark.

intermediate

3 hours 3 mins

Course

Introduction to Big Data and Hadoop

beginner

10 hour

Course

An Introduction to Spark

beginner

2 hour 50 min

Course

From Pandas to PySpark DataFrame

intermediate

3 hour 3 min

Related Courses and Skill Paths

Course

Mastering Apache Kafka

Gain insights into Apache Kafka’s architecture, Client APIs, stream processing, and connectors. Discover hands-on techniques for building real-time data streaming and event-driven applications. Explore Kafka's core ecosystem.

intermediate

7 hours

Course

Data Science in Production: Building Scalable Model Pipelines

Gain insights into building scalable data and model pipelines, explore different cloud environments, delve into streaming workflows, and discover essential tools for creating real-time data products.

advanced

8 hours

Course

Cracking the Google Associate Cloud Engineer Certification

Become a cloud pro with strategic GCP skills. Get hands-on experience, understand "Why GCP?", and prepare with a practice exam. No setup, no cleanup, no hassle.

beginner

4 hours

Course

Mastering Apache Kafka

intermediate

7 hour

Course

Data Science in Production: Building Scalable Model Pipelines

advanced

8 hour

Course

Cracking the Google Associate Cloud Engineer Certification