Home>Courses>Transferring Data with ETL

Transferring Data with ETL

Delve into ETL processes to transfer data. Learn to extract, transform, and load using SQL, Python, and Apache Spark. Explore automation with Apache Airflow and pandas. Gain business intelligence insights.

Intermediate

57 Lessons

10h

Certificate of Completion

AI-POWERED

Explanations

AI-POWERED

Explanations

This course includes

1 Project

133 Playgrounds

7 Quizzes

Course Overview

What You'll Learn

Course Content

Recommendations

Course Overview

ETL stands for extract, transform, and load. It’s a collection of processes that combine data from various sources and load them into data warehouses or other data repositories. ETL is crucial for providing data used for business intelligence and analytics. In this course, you’ll experiment with extracting data from various database solutions such as MySQL, PostgreSQL, and MongoDB. You’ll use query and scripting languages like SQL, Python, and Apache Spark to process data and load it to data repositories o...Show More

What You'll Learn

An understanding of the extract, transform, and load steps in an ETL pipeline

Hands-on experience implementing and orchestrating the ETL pipelines

Understanding of databases, data warehousing, data processing, and data ingestion

Hands-on experience with ETL tools such as Python, SQL, Apache Spark, and Apache Airflow

An understanding of the extract, transform, and load steps in an ETL pipeline

Course Content

Introduction

14 Lessons

Get familiar with building ETL pipelines, their stages, and practical data transformation examples.

Getting Started

ETL Pipeline Stages

What Is an ETL Pipeline?

A New Paradigm—ELT

ETL Example—Extraction

ETL Transformation Example: Addressing Data Quality Issue

ETL Transformation Example: Handling Missing Values and Data

ETL Transformation Example: Sorting and Finalizing the Data

ETL Example—Load

ETL Example—Scheduling

Batch vs. Stream Processing

Data Warehouse

Examples and Use Cases

Quiz: ETL Pipelines

E: Extract

14 Lessons

Get started with techniques for data extraction from various sources including databases, APIs, and web scraping.

Introduction

Data Extraction Methods Overview

Extracting Data with Web Scraping

Web Scraping Exercise: Reading the Data

Web Scraping Exercise: DataFrames to CSV

Extraction Using a REST API

Exercise: Extracting Data with a REST API

Full Extraction From MySQL Database

Incremental Extraction From MySQL Database

Extraction From MySQL’s Binary Log

Extract From PostgreSQL Database

Extraction From Google BigQuery

Exercise: Databases

Quiz: Extracting Data

T: Transform

11 Lessons

Master the steps to transform raw data into usable formats through cleaning, structuring, anonymizing, and aggregating.

Introduction

Data Typing and Structuring Using Python

Exercise: Data Typing and Structuring

Anonymizing and Encrypting Using Python

Exercise: Anonymizing and Encrypting the Data

Data Cleaning Using Apache Spark: Missing Data

Data Cleaning Using Apache Spark: Duplicate Data

Exercise: Data Cleaning

Filtering, Sorting, Aggregating, and Binning Using SQL

Exercise: Filtering, Sorting, Aggregating, and Binning

Quiz: Transforming Data

L: Load

9 Lessons

Grasp the fundamentals of loading transformed data into repositories, hosting options, and loading strategies.

Introduction

Hosting and Deployment: On-Premise vs. Cloud-Based

Hosting and Deployment: Open-Sourced vs. Proprietary

Loading Methods: Scheduled vs. On-Demand

Loading Methods: Full Loading vs. Incremental Loading

Loading Data Using the COPY Command

Exercise: The COPY Command

Exercise: Incremental Load

Quiz: Loading Data

Orchestration

8 Lessons

Take a closer look at orchestrating ETL pipelines with Apache Airflow, deployment, and task management.

Introduction

Deploying Airflow

ETL Pipeline Exercise: Extracting Data

ETL Pipeline Example: Airflow Extraction Task

ETL Pipeline Exercise: Transform

ETL Pipeline Example: Airflow Transform Task

ETL Pipeline Example: Load

Interacting with Airflow

ETL Pipeline: Fraud Detection Preprocessing

Project

Conclusion

1 Lessons

Implement essential ETL skills and tools for effective data architecture and automation.

Final Thoughts

Trusted by 2.5 million developers working at companies

"These are high-quality courses. Trust me. I own around 10 and the price is worth it for the content quality. EducativeInc came at the right time in my career. I'm understanding topics better than with any book or online video tutorial I've done. Truly made for developers. Thanks"

Anthony Walker

@_webarchitect_

"Just finished my first full #ML course: Machine learning for Software Engineers from Educative, Inc. ... Highly recommend!"

Evan Dunbar

ML Engineer

"You guys are the gold standard of crash-courses... Narrow enough that it doesn't need years of study or a full blown book to get the gist, but broad enough that an afternoon of Googling doesn't cut it."

Carlos Matias La Borde

Software Developer

"I spend my days and nights on Educative. It is indispensable. It is such a unique and reader-friendly site"

Souvik Kundu

Front-end Developer

"Your courses are simply awesome, the depth they go into and the breadth of coverage is so good that I don't have to refer to 10 different websites looking for interview topics and content."

Vinay Krishnaiah

Software Developer

"I've tried probably 5-7 different sites and Educative is easily the best. It perfectly blends explanation with interactivity"

Eric Downs

Musician/Entrepeneur

Hands-on Learning Powered by AI

See how Educative uses AI to make your learning more immersive than ever before.

Instant Code Feedback

Evaluate and debug your code with the click of a button. Get real-time feedback on test cases, including time and space complexity of your solutions.

AI-Powered Mock Interviews

Put your skills to the test in a simulated interview setting. Receive personalized feedback based on your performance. Available in Premium & Premium Plus plans.

Adaptive Learning

At various checkpoints throughout Educative courses, you will be prompted to take a quick assessment. Receive a condensed curriculum tailored to your strengths and skill gaps.

Explain with AI

Select any text within any Educative course, and get an instant explanation — without ever leaving your browser.

AI Code Mentor

AI Code Mentor helps you quickly identify errors in your code, learn from your mistakes, and nudge you in the right direction — just like a 1:1 tutor!

Recommended before starting this course

Course

Getting Started with SQL and Relational Databases

Gain insights into using SQL for relational databases. Learn about creating, editing, and managing tables with MySQL, creating indexes, complex queries, and joining tables for optimized data retrieval.

beginner

9 hours 40 mins

Course

Predictive Data Analysis with Python

Gain insights into performing predictive data analysis using Python. Learn about tools like NumPy, Pandas, Matplotlib, and Seaborn, and applying techniques to real-world financial and advertisement projects.

beginner

10 hours

Course

Python Data Analysis and Visualization

Delve into Python data analysis and visualization using NumPy, SciPy, Pandas, and Seaborn. Gain insights into reading, cleaning, and visualizing data to generate actionable insights.

beginner

2 hours 30 mins

Course

Getting Started with SQL and Relational Databases

beginner

9 hour 40 min

Course

Predictive Data Analysis with Python

beginner

10 hour

Course

Python Data Analysis and Visualization

beginner

2 hour 30 min

Related Courses and Skill Paths

Course

Data Science in Production: Building Scalable Model Pipelines

Gain insights into building scalable data and model pipelines, explore different cloud environments, delve into streaming workflows, and discover essential tools for creating real-time data products.

advanced

8 hours

Course

Mastering Big Data with PySpark

Gain insights into PySpark within big data. Learn about data ingestion, distributed computing, data processing, and performance optimization to solve real-world problems and apply machine learning.

beginner

12 hours

Course

Mastering Big Data with Apache Spark and Java

Gain insights into Spark Java API, learn about data transformations and SQL operations, and discover how to integrate big data and Java for scalable, high-speed processing.

intermediate

17 hours 15 mins

Course

Data Science in Production: Building Scalable Model Pipelines

advanced

8 hour

Course

Mastering Big Data with PySpark

beginner

12 hour

Course

Mastering Big Data with Apache Spark and Java