Building a Machine Learning Pipeline from Scratch

Gain insights into ML pipeline development, delve into best practices, discover advanced Python concepts, and explore testing methodologies to elevate your software engineering skills and career prospects.

Beginner

42 Lessons

14h

Certificate of Completion

AI-POWERED

Explanations

AI-POWERED

Explanations

This course includes

1 Assessment

30 Playgrounds

7 Quizzes

This course includes

1 Assessment

30 Playgrounds

7 Quizzes

Course Overview

Machine learning (ML) has matured into a mainstream development activity, and data scientists are expected to be able to write production-grade training pipelines. This course will provide you with a foundation in ML pipeline development guided by best practices in software engineering. You’ll start by learning about code organization, style, and conceptual ideas, such as topological sorting of directed acyclic graphs. You’ll dive into the hands-on development of an ML pipeline. You’ll learn some advanced ...Show More

TAKEAWAY SKILLS

Python

Machine Learning

Data Science

Data Pipeline Engineering

Machine Learning Paradigms

Python 3

Python Programming

Unit Testing

Machine Learning Fundamentals

Data Manipulation

What You'll Learn

An understanding of what constitutes as a machine learning training pipeline

Hands-on experience building a machine learning pipeline in Python

Familiarity with advanced Python concepts, such as abstract base classes and mixins

An understanding of software engineering best practices, including code style, documentation, and logging

A working knowledge of unit testing in Python

What You'll Learn

An understanding of what constitutes as a machine learning training pipeline

Course Content

Introduction

Get familiar with building and deploying a machine learning pipeline from scratch.

Course Goals and Structure

Getting Started

Look at how traditional engineering practices enhance ML pipeline stability and collaboration.

Motivation: Why Jupyter Notebooks Aren’t Enough

The ML Training Pipeline

Quiz: Getting Started

Structuring the ML Pipeline

Break apart the ML pipeline structure, directory organization, code style, and dependency management.

Managing Dependencies

Quiz: Structuring the ML Pipeline

Directed Acyclic Graphs (DAGs)

Break down the steps to construct and sort DAGs for machine learning tasks.

What Is a DAG?

Topological Sorting

Quiz: DAGs

The ML Library

Unveil object-oriented implementation, configuration management, dataset and model handling, and report generation in ML pipelines.

Object-Oriented Programming

Configuration Files: OmegaConf

Abstract Base Classes

The Pipeline Core

7 Lessons

Follow the process of structuring machine learning pipelines, including argument parsing, logging, and tracking experiments.

Extending the Pipeline

2 Lessons

Build on extending machine learning pipelines to support new datasets and models effectively.

Testing

4 Lessons

Step through unit testing, Pytest for code coverage, and comprehensive system testing.

Deployment

2 Lessons

Get started with packaging and deploying machine learning models for consistent predictions.

10.

Other Considerations

4 Lessons

Explore considerations for data quality monitoring, reproducibility, and leveraging off-the-shelf ML solutions.

11.

Wrapping Up

1 Lesson

Grasp the fundamentals of building, managing, and deploying a machine learning pipeline.

12.

Appendix

1 Lesson

Take a closer look at essential resources for machine learning pipelines, libraries, and frameworks.

Final Assessment

Assessment

Course Author

Trusted by 1.4 million developers working at companies

Anthony Walker

@_webarchitect_

Evan Dunbar

ML Engineer

Carlos Matias La Borde

Software Developer

Souvik Kundu

Front-end Developer

Vinay Krishnaiah

Software Developer

Eric Downs

Musician/Entrepeneur

Kenan Eyvazov

DevOps Engineer

Souvik Kundu

Front-end Developer

Eric Downs

Musician/Entrepeneur

Anthony Walker

@_webarchitect_

Evan Dunbar

ML Engineer

Hands-on Learning Powered by AI

See how Educative uses AI to make your learning more immersive than ever before.

Instant Code Feedback

Evaluate and debug your code with the click of a button. Get real-time feedback on test cases, including time and space complexity of your solutions.

AI-Powered Mock Interviews

Put your skills to the test in a simulated interview setting. Receive personalized feedback based on your performance. Available in Premium & Premium Plus plans.

Adaptive Learning

At various checkpoints throughout Educative courses, you will be prompted to take a quick assessment. Receive a condensed curriculum tailored to your strengths and skill gaps.

Explain with AI

Select any text within any Educative course, and get an instant explanation — without ever leaving your browser.

AI Code Mentor

AI Code Mentor helps you quickly identify errors in your code, learn from your mistakes, and nudge you in the right direction — just like a 1:1 tutor!

FOR TEAMS

Interested in this course for your business or team?

Unlock this course (and 1,000+ more) for your entire org with DevPath