HomeCoursesBuilding a Machine Learning Pipeline from Scratch

4.4

Beginner

14h

Updated 2 months ago

Building a Machine Learning Pipeline from Scratch

Learn about ML pipeline development, delve into best practices, discover advanced Python concepts, and explore testing methodologies to elevate your software engineering skills and career prospects.

Join 2.7 million developers at

Overview

Content

Reviews

Machine learning (ML) has matured into a mainstream development activity, and data scientists are expected to be able to write production-grade training pipelines. This course will provide you with a foundation in ML pipeline development guided by best practices in software engineering. You’ll start by learning about code organization, style, and conceptual ideas, such as topological sorting of directed acyclic graphs. You’ll dive into the hands-on development of an ML pipeline. You’ll learn some advanced Python concepts and useful libraries. Finally, you’ll learn about testing methodologies and how to package your code into a distributable library. By the end of this course, you’ll understand what makes a great ML pipeline work, better appreciate software engineering processes, improve your Python knowledge, and learn how to build ML training pipelines. This knowledge will make you more attractive to potential employers and improve your chances of thriving as a new data scientist.

Machine learning (ML) has matured into a mainstream development activity, and data scientists are expected to be able to write p...Show More

WHAT YOU'LL LEARN

An understanding of what constitutes as a machine learning training pipeline

Hands-on experience building a machine learning pipeline in Python

Familiarity with advanced Python concepts, such as abstract base classes and mixins

An understanding of software engineering best practices, including code style, documentation, and logging

A working knowledge of unit testing in Python

An understanding of what constitutes as a machine learning training pipeline

TAKEAWAY SKILLS

Python

Machine Learning

Data Science

Data Pipeline Engineering

Machine learning paradigms

Python 3

Python Programming

Unit Testing

Machine Learning Fundamentals

Machine learning fundamentals

Data Manipulation

Content

42 Lessons7 Quizzes1 Assessment

Introduction

1 Lessons

Get familiar with building and deploying a machine learning pipeline from scratch.

Course Goals and Structure

Getting Started

3 Lessons

Look at how traditional engineering practices enhance ML pipeline stability and collaboration.

Motivation: Why Jupyter Notebooks Aren’t Enough

The ML Training Pipeline

Quiz: Getting Started

Structuring the ML Pipeline

6 Lessons

Break apart the ML pipeline structure, directory organization, code style, and dependency management.

Managing Dependencies

Quiz: Structuring the ML Pipeline

Directed Acyclic Graphs (DAGs)

3 Lessons

Break down the steps to construct and sort DAGs for machine learning tasks.

What Is a DAG?

Topological Sorting

Quiz: DAGs

The ML Library

8 Lessons

Unveil object-oriented implementation, configuration management, dataset and model handling, and report generation in ML pipelines.

Object-Oriented Programming

Configuration Files: OmegaConf

Abstract Base Classes

Create Your First Data Pipeline with a Dashboard

Project

The Pipeline Core

7 Lessons

Follow the process of structuring machine learning pipelines, including argument parsing, logging, and tracking experiments.

Structuring the Pipeline Core

Parsing Command-Line Arguments: Argparse

Experiment Tracking

Logging

The Complete Pipeline

Docstrings

Quiz: The Pipeline Core

Extending the Pipeline

2 Lessons

Build on extending machine learning pipelines to support new datasets and models effectively.

Supporting a New Dataset

Supporting a New Model

Build a News ETL Data Pipeline Using Python and SQLite

Project

Testing

4 Lessons

Step through unit testing, Pytest for code coverage, and comprehensive system testing.

Introduction to Unit Testing

Pytest

Introduction to System Testing

Quiz: Testing

Deployment

2 Lessons

Get started with packaging and deploying machine learning models for consistent predictions.

Packaging the ML Library

Inference

10.

Other Considerations

4 Lessons

Explore considerations for data quality monitoring, reproducibility, and leveraging off-the-shelf ML solutions.

Data Quality and Model Performance Monitoring

Reproducibility

Off-the-Shelf Solutions

Quiz: Observability and Reproducibility

11.

Wrapping Up

1 Lessons

Grasp the fundamentals of building, managing, and deploying a machine learning pipeline.

Conclusion

12.

Appendix

1 Lessons

Take a closer look at essential resources for machine learning pipelines, libraries, and frameworks.

Useful Resources

Final Assessment

Assessment

Certificate of Completion

Showcase your accomplishment by sharing your certificate of completion.

Course Author:

Jayanth Chennamangalam

Developed by MAANG Engineers

Every Educative resource is designed by our team of ex-MAANG software engineers and PhD computer science educators — subject matter experts who’ve shipped production code at scale and taught the theory behind it. The goal is to get you hands-on with the skills you need to stay ahead in today's constantly evolving tech landscape. No videos, no fluff — just interactive, project-based learning with personalized feedback that adapts to your goals and experience.

Trusted by 2.7 million developers working at companies

"These are high-quality courses. Trust me. I own around 10 and the price is worth it for the content quality. EducativeInc came at the right time in my career. I'm understanding topics better than with any book or online video tutorial I've done. Truly made for developers. Thanks"

Anthony Walker

@_webarchitect_

"Just finished my first full #ML course: Machine learning for Software Engineers from Educative, Inc. ... Highly recommend!"

Evan Dunbar

ML Engineer

"You guys are the gold standard of crash-courses... Narrow enough that it doesn't need years of study or a full blown book to get the gist, but broad enough that an afternoon of Googling doesn't cut it."

Software Developer

Carlos Matias La Borde

"I spend my days and nights on Educative. It is indispensable. It is such a unique and reader-friendly site"

Souvik Kundu

Front-end Developer

"Your courses are simply awesome, the depth they go into and the breadth of coverage is so good that I don't have to refer to 10 different websites looking for interview topics and content."

Vinay Krishnaiah

Software Developer

Hands-on Learning Powered by AI

See how Educative uses AI to make your learning more immersive than ever before.

Personalized Interview Prep

Skip the LeetCode grind with a custom roadmap that adapts to your goals. Hands-on practice for Coding Interviews, System Design, and more.

Mock Interviews

Test your skills in a simulated interview setting. Receive personalized feedback based on your performance. Available for Coding Interviews, System Design, and more.

AI Prompt

Build prompt engineering skills. Practice implementing AI-informed solutions.

Code Feedback

Evaluate and debug your code with the click of a button. Get real-time feedback on test cases, including time and space complexity of your solutions.

Explain with AI

Select any text within any Educative course, and get an instant explanation — without ever leaving your browser.

AI Code Mentor

AI Code Mentor helps you quickly identify errors in your code, learn from your mistakes, and nudge you in the right direction — just like a 1:1 tutor!

Course

Interactive Dashboards and Data Apps with Plotly and Dash

Gain insights into creating interactive dashboards and data apps with Plotly Dash. Learn to build, customize, and deploy data-driven applications with various controls and charts effectively.

27 h

beginner

Course

Using Python Altair for Data Storytelling

Gain insights into using Python Altair for data storytelling. Learn about the DIKW pyramid and craft interactive visualizations to effectively convey complex ideas and narratives through data.

17 h

intermediate

Course

Data Storytelling through Visualizations in Python

Gain insights into data storytelling in Python using Matplotlib, Seaborn, and Plotly. Explore trends, handle data challenges, and create impactful visual narratives aligning with business goals.

9 h

intermediate

Course

Learn to Code: Python for Absolute Beginners

Want to learn how to code? Get hands-on with Python for beginners. Master the basics, solve real-world problems, and confidently build in-demand Python skills for diverse industries.

8 h

beginner

Course

Python 101: Interactively learn how to program with Python 3

This is a 101-level course in Python that will help you learn to program in Python for free.

10 h

beginner

Course

Python Data Analysis and Visualization

Delve into Python data analysis and visualization using NumPy, SciPy, Pandas, and Seaborn. Gain insights into reading, cleaning, and visualizing data to generate actionable insights.

2 h 30 m

beginner

Course

Interactive Dashboards and Data Apps with Plotly and Dash

beginner

27 hour

Course

Using Python Altair for Data Storytelling

intermediate

17 hour

Course

Data Storytelling through Visualizations in Python