gcp_auth

application_default_credentials.json from GCP SDK installation

project_id

Gain insights into data engineering foundations, explore data life cycle stages, and delve into creating data pipelines using Python, Kafka, PySpark, Airflow, and dbt.

UDoc.tar.gz

BigQuery

Airflow

Airflow Custom

Dagster

Airflow-copy

Airflow-custom-copy

Data engineering is currently one of the most in-demand fields in data and technology. It intersects software engineering, DataOps, data architecture, data management, and security. Data engineers, such as analysts and data scientists, lay the foundation to serve data for consumers. 

In this course, you will learn the foundation of data engineering, covering different parts of the entire data life cycle: data warehouse, ingestion, transformation, orchestration, etc. You will also gain hands-on experience building data pipelines using different techniques such as Python, Kafka, PySpark, Airflow, dbt, and more. 

By the end of this course, you will have a holistic understanding of data engineering and be able to build your data pipelines to serve data for various consumers.

Data Engineering Foundations in Python

A data lake is a popular data architecture comparable, to a data warehouse. It's a storage repository that holds a large amount of data, but unlike a data warehouse where data is structured, data in a data lake is in its raw format. Apart from the format, the following table summarizes other differences:

Data Lake vs. Data Warehouse

align-center

Understand two types of cloud data architectures: data lake and data mesh.

Topic	Data Lake	Data Warehouse
Data Format	Store unstructured, semi-structured and structured data in its raw format.	Store only structured data after the transformation.
Schema	Schema-on-read: Schema is defined after data is stored.	Schema-on-write: Schema is predefined prior to when data is stored.
Usecase	Data exploration: Unstructured data opens more possibilities for analysis and ML algorithms. A landing place before loading data into a data warehouse.	Reporting: Reporting tools and dashboards prefer highly coherent data.
Data Quality	Data is in its raw format without cleaning, so data quality is not ensured.	Data is highly curated, resulting in higher data quality.
Cost	Both storage and operational costs are lower.	Storing data in the data warehouse is usually more expensive and time-consuming.

Getting Started

Data Team Structure

Data Engineering Life Cycle

Cloud Data Architecture

Data Ingestion

Data Modeling

Data Orchestration

Mastering Airflow: Building an ETL Pipeline

Data Quality

Build an End-to-End Data Pipeline for Formula 1 Analysis

Epilogue

Appendix

Cloud Data Architectures: Data Lake and Data Mesh

Data lake

Data Lake vs. Data Warehouse