gcp_auth

application_default_credentials.json from GCP SDK installation

project_id

Gain insights into data engineering foundations, explore data life cycle stages, and delve into creating data pipelines using Python, Kafka, PySpark, Airflow, and dbt.

UDoc.tar.gz

BigQuery

Airflow

Airflow Custom

Dagster

Airflow-copy

Airflow-custom-copy

Data engineering is currently one of the most in-demand fields in data and technology. It intersects software engineering, DataOps, data architecture, data management, and security. Data engineers, such as analysts and data scientists, lay the foundation to serve data for consumers. 

In this course, you will learn the foundation of data engineering, covering different parts of the entire data life cycle: data warehouse, ingestion, transformation, orchestration, etc. You will also gain hands-on experience building data pipelines using different techniques such as Python, Kafka, PySpark, Airflow, dbt, and more. 

By the end of this course, you will have a holistic understanding of data engineering and be able to build your data pipelines to serve data for various consumers.

Data Engineering Foundations in Python

**(Select all that apply.)** In which service models does the provider provide middleware?

(Select all that apply.) In which service models does the provider provide middleware?

Which service model offloads the most responsibilities to the service provider?

Which service model offloads the most responsibilities to the service provider?

Lambda architecture handles both batch and streaming data.

Lambda architecture can be challenging to maintain due to two parallel systems.

Lambda architecture persists data in a primary database.

Lambda architecture only reads streaming data.

Which statement is incorrect about Lambda architecture?

Which statement is incorrect about Lambda architecture?

Kappa architecture only reads streaming data.

Kappa architecture doesn't persist data anywhere.

Kappa architecture was invented after Lambda architecture.

Kappa architecture is mainly used in event-based architecture.

Which statement is incorrect about Kappa architecture?

Which statement is incorrect about Kappa architecture?

Which cloud architecture uses schema-on-read?

Which cloud architecture uses schema-on-read?

What are the advantages of cloud architecture?

What are the advantages of cloud architecture?

Reduced flexibility in resource allocation

**(Select all that apply.)** What are the disadvantages of cloud architecture?

(Select all that apply.) What are the disadvantages of cloud architecture?

Test your knowledge of cloud data architecture.

Getting Started

Data Team Structure

Data Engineering Life Cycle

Cloud Data Architecture

Data Ingestion

Data Modeling

Data Orchestration

Mastering Airflow: Building an ETL Pipeline

Data Quality

Build an End-to-End Data Pipeline for Formula 1 Analysis

Epilogue

Appendix

Quiz: Cloud Data Architecture