gcp_auth

application_default_credentials.json from GCP SDK installation

project_id

This course is about data engineering using GCP, Python, Kafka, PySpark, and Airflow, covering the entire data lifecycle from ingestion to transformation.

UDoc.tar.gz

BigQuery

Airflow

Airflow Custom

Dagster

Airflow-copy

Airflow-custom-copy

Data engineering is currently one of the most in-demand fields in data and technology. It intersects software engineering, DataOps, data architecture, data management, and security. Data engineers, such as analysts and data scientists, lay the foundation to serve data for consumers. 

In this course, you will learn the foundation of data engineering, covering different parts of the entire data life cycle: data warehouse, ingestion, transformation, orchestration, etc. You will also gain hands-on experience building data pipelines using different techniques such as Python, Kafka, PySpark, Airflow, dbt, and more. 

By the end of this course, you will have a holistic understanding of data engineering and be able to build your data pipelines to serve data for various consumers.

Data Engineering Foundations in Python

Relational, object-oriented, and document-oriented

Normalized, denormalized, and semi-structured

What are the different types of data models?

What are the different types of data models?

A model that represents the relationships between entities in a database.

A model that represents the structure of a database at the lowest level.

A model that represents the overall structure of a database, independent of any specific technology or implementation.

A model that represents the physical implementation of a database.

Which statement best describes the logical data model?

Which statement best describes the logical data model?

All types of data models are equally abstract

Which type of data model is the most abstract?

Which type of data model is the most abstract?

Both types of databases can be used to support business operations

Neither type of database is used to support business operations

Which type of database is typically used to support business operations such as online ordering or inventory management?

Which type of database is typically used to support business operations such as online ordering or inventory management?

Both types of databases are used to support business intelligence and decision-making

Neither type of database is used to support business intelligence and decision-making

Which type of database is typically used to support business intelligence and decision-making?

Which type of database is typically used to support business intelligence and decision-making?

A table that contains the attributes of the entities.

A table that contains the metrics or measurements.

A table that contains the relationships between the entities.

A table that contains metadata about the data being analyzed.

What is a fact table in Kimball dimensional modeling?

What is a fact table in Kimball dimensional modeling?

A schema that has multiple fact tables and a single dimension table.

A schema that has multiple fact tables and multiple dimension tables.

A schema that has a single fact table and multiple dimension tables.

A schema that does not have fact tables or dimension tables.

Which statement best describes the star schema in Kimball dimensional modeling?

Which statement best describes the star schema in Kimball dimensional modeling?

By overwriting the existing row in the dimension table with the latest attribute value.

By adding a new column to the fact table for each change in the dimension.

By creating a separate mini-dimension table for fast-changing dimensions.

By adding a new row to the dimension table for each change in the dimension with the effective period.

How is an SCD type 2 table modeled in a Kimball dimensional model?

How is an SCD type 2 table modeled in a Kimball dimensional model?

Getting Started

Data Team Structure

Data Engineering Life Cycle

Cloud Data Architecture

Data Ingestion

Data Modeling

Data Orchestration

Data Quality

Build an End-to-End Data Pipeline for Formula 1 Analysis

Epilogue

Appendix

Quiz: Data Modeling