Data Engineering Foundations in Python

This course is about data engineering using GCP, Python, Kafka, PySpark, and Airflow, covering the entire data lifecycle from ingestion to transformation.

Beginner

46 Lessons

Certificate of Completion

This course is about data engineering using GCP, Python, Kafka, PySpark, and Airflow, covering the entire data lifecycle from ingestion to transformation.

AI-POWERED

Explanations

AI-POWERED

Explanations

This course includes

1 Project

57 Playgrounds

7 Quizzes

This course includes

1 Project

57 Playgrounds

7 Quizzes

Course Overview

Data engineering is currently one of the most in-demand fields in data and technology. It intersects software engineering, DataOps, data architecture, data management, and security. Data engineers, such as analysts and data scientists, lay the foundation to serve data for consumers. In this course, you will learn the foundation of data engineering, covering different parts of the entire data life cycle: data warehouse, ingestion, transformation, orchestration, etc. You will also gain hands-on experience b...Show More

TAKEAWAY SKILLS

Python

Sql

What You'll Learn

An understanding of the data engineering life cycle

Familiarity with the cloud data warehouse and data modeling techniques

Hands-on experience with data engineering tools such as GCP, Airflow, Spark, and dbt

The ability to build data pipelines from scratch in Python

What You'll Learn

An understanding of the data engineering life cycle

Course Content

Getting Started

Introduction

Data Team Structure

Data Team Roles

Data Team Structure: Embedded or Centralized?Quiz: Data Team Structure

Data Engineering Life Cycle

Set Up the Google Cloud Introduction to Data Engineering Life Cycle Ingestion, Transformation, and Visualization Storage and Infrastructure Quiz: Data Engineering Life Cycle

Cloud Data Architecture

Evolution of Cloud Data Architecture Cloud Service Models Cloud Data Architectures: Lambda, Kappa, and Data Warehouse Cloud Data Architectures: Data Lake and Data Mesh Patterns of Good Cloud Data Architecture Quiz: Cloud Data Architecture

Data Ingestion

Batch Ingestion Stream Ingestion Push vs. Pull Ingestion Methods—SFTP, API, and Object Storage Ingestion Methods—CDC Ingestion Methods—Streaming Platform Ingest with pandas Ingest with PySpark Quiz: Data Ingestion

Data Modeling

8 Lessons

Data Orchestration

8 Lessons

Data Quality

4 Lessons

Build an End-to-End Data Pipeline for Formula 1 Analysis

Project

Epilogue

1 Lesson

10.

Appendix

1 Lesson

Course Author

Trusted by 1.4 million developers working at companies

Anthony Walker

@_webarchitect_

Emma Bostian 🐞

@EmmaBostian

Evan Dunbar

ML Engineer

Carlos Matias La Borde

Software Developer

Souvik Kundu

Front-end Developer

Vinay Krishnaiah

Software Developer

Eric Downs

Musician/Entrepeneur

Kenan Eyvazov

DevOps Engineer

Anthony Walker

@_webarchitect_

Emma Bostian 🐞

@EmmaBostian

Hands-on Learning Powered by AI

See how Educative uses AI to make your learning more immersive than ever before.

Instant Code Feedback

Evaluate and debug your code with the click of a button. Get real-time feedback on test cases, including time and space complexity of your solutions.

AI-Powered Mock Interviews

Put your skills to the test in a simulated interview setting. Receive personalized feedback based on your performance. Available in Premium & Premium Plus plans.

Adaptive Learning

At various checkpoints throughout Educative courses, you will be prompted to take a quick assessment. Receive a condensed curriculum tailored to your strengths and skill gaps.

Explain with AI

Select any text within any Educative course, and get an instant explanation — without ever leaving your browser.

AI Code Mentor

AI Code Mentor helps you quickly identify errors in your code, learn from your mistakes, and nudge you in the right direction — just like a 1:1 tutor!

FOR TEAMS

Interested in this course for your business or team?

Unlock this course (and 1,000+ more) for your entire org with DevPath