gcp_auth

application_default_credentials.json from GCP SDK installation

project_id

Gain insights into data engineering foundations, explore data life cycle stages, and delve into creating data pipelines using Python, Kafka, PySpark, Airflow, and dbt.

UDoc.tar.gz

BigQuery

Airflow

Airflow Custom

Dagster

Airflow-copy

Airflow-custom-copy

Data engineering is currently one of the most in-demand fields in data and technology. It intersects software engineering, DataOps, data architecture, data management, and security. Data engineers, such as analysts and data scientists, lay the foundation to serve data for consumers. 

In this course, you will learn the foundation of data engineering, covering different parts of the entire data life cycle: data warehouse, ingestion, transformation, orchestration, etc. You will also gain hands-on experience building data pipelines using different techniques such as Python, Kafka, PySpark, Airflow, dbt, and more. 

By the end of this course, you will have a holistic understanding of data engineering and be able to build your data pipelines to serve data for various consumers.

Data Engineering Foundations in Python

Which is *not* one of the dimensions of data quality?
 

Which is not one of the dimensions of data quality?

High data quality is not essential for making good decisions.

Poor data quality can lead to incorrect decisions and outcomes.

Data quality only affects operational processes, not decision-making.

Data quality has no impact on business outcomes.

How does data quality impact business decisions and outcomes?


How does data quality impact business decisions and outcomes?

A data schema is a type of data analysis technique.

A data schema is a technique for structuring and formatting data.

Which statement about data schema is *true*?


Which statement about data schema is true?

Compared to Avro, what is one of the advantages of using Protocol Buffers?

Compared to Avro, what is one of the advantages of using Protocol Buffers?

Unit tests run before deploying SQL change to production, while data tests run during the production execution. 

Unit tests normally use a prepared dataset while data test uses the production data.

Unit tests and data tests have the same purposes.

Both unit tests and data tests can help ensure the data quality.

Which statement is *not* correct about data test and unit test?

Which statement is not correct about data test and unit test?

Test your knowledge of data quality covered in this chapter.

Getting Started

Data Team Structure

Data Engineering Life Cycle

Cloud Data Architecture

Data Ingestion

Data Modeling

Data Orchestration

Mastering Airflow: Building an ETL Pipeline

Data Quality

Build an End-to-End Data Pipeline for Formula 1 Analysis

Epilogue

Appendix

Quiz: Data Quality