Home>Courses>Building a Machine Learning Pipeline from Scratch

Building a Machine Learning Pipeline from Scratch

Gain insights into ML pipeline development, delve into best practices, discover advanced Python concepts, and explore testing methodologies to elevate your software engineering skills and career prospects.

Beginner

42 Lessons

14h

Certificate of Completion

AI-POWERED

Explanations

AI-POWERED

Explanations

This course includes

1 Assessment

30 Playgrounds

7 Quizzes

Course Overview

What You'll Learn

Course Content

Recommendations

Course Overview

Machine learning (ML) has matured into a mainstream development activity, and data scientists are expected to be able to write production-grade training pipelines. This course will provide you with a foundation in ML pipeline development guided by best practices in software engineering. You’ll start by learning about code organization, style, and conceptual ideas, such as topological sorting of directed acyclic graphs. You’ll dive into the hands-on development of an ML pipeline. You’ll learn some advanced ...Show More

Machine learning (ML) has matured into a mainstream development activity, and data scientists are expected to be able to write p...Show More

TAKEAWAY SKILLS

Python

Machine Learning

Data Science

Data Pipeline Engineering

Machine learning paradigms

Python 3

Python Programming

Unit Testing

Machine Learning Fundamentals

Machine learning fundamentals

Data Manipulation

What You'll Learn

An understanding of what constitutes as a machine learning training pipeline

Hands-on experience building a machine learning pipeline in Python

Familiarity with advanced Python concepts, such as abstract base classes and mixins

An understanding of software engineering best practices, including code style, documentation, and logging

A working knowledge of unit testing in Python

An understanding of what constitutes as a machine learning training pipeline

Course Content

Introduction

1 Lessons

Get familiar with building and deploying a machine learning pipeline from scratch.

Course Goals and Structure

Getting Started

3 Lessons

Look at how traditional engineering practices enhance ML pipeline stability and collaboration.

Motivation: Why Jupyter Notebooks Aren’t Enough

The ML Training Pipeline

Quiz: Getting Started

Structuring the ML Pipeline

6 Lessons

Break apart the ML pipeline structure, directory organization, code style, and dependency management.

Managing Dependencies

Quiz: Structuring the ML Pipeline

Directed Acyclic Graphs (DAGs)

3 Lessons

Break down the steps to construct and sort DAGs for machine learning tasks.

What Is a DAG?

Topological Sorting

Quiz: DAGs

The ML Library

8 Lessons

Unveil object-oriented implementation, configuration management, dataset and model handling, and report generation in ML pipelines.

Object-Oriented Programming

Configuration Files: OmegaConf

Abstract Base Classes

The Pipeline Core

7 Lessons

Follow the process of structuring machine learning pipelines, including argument parsing, logging, and tracking experiments.

Structuring the Pipeline Core

Parsing Command-Line Arguments: Argparse

Experiment Tracking

Logging

The Complete Pipeline

Docstrings

Quiz: The Pipeline Core

Extending the Pipeline

2 Lessons

Build on extending machine learning pipelines to support new datasets and models effectively.

Supporting a New Dataset

Supporting a New Model

Testing

4 Lessons

Step through unit testing, Pytest for code coverage, and comprehensive system testing.

Introduction to Unit Testing

Pytest

Introduction to System Testing

Quiz: Testing

Deployment

2 Lessons

Get started with packaging and deploying machine learning models for consistent predictions.

Packaging the ML Library

Inference

10.

Other Considerations

4 Lessons

Explore considerations for data quality monitoring, reproducibility, and leveraging off-the-shelf ML solutions.

Data Quality and Model Performance Monitoring

Reproducibility

Off-the-Shelf Solutions

Quiz: Observability and Reproducibility

11.

Wrapping Up

1 Lessons

Grasp the fundamentals of building, managing, and deploying a machine learning pipeline.

Conclusion

12.

Appendix

1 Lessons

Take a closer look at essential resources for machine learning pipelines, libraries, and frameworks.

Useful Resources

Final Assessment

Assessment

Course Author

Jayanth Chennamangalam

Trusted by 2.6 million developers working at companies

"These are high-quality courses. Trust me. I own around 10 and the price is worth it for the content quality. EducativeInc came at the right time in my career. I'm understanding topics better than with any book or online video tutorial I've done. Truly made for developers. Thanks"

Anthony Walker

@_webarchitect_

"Just finished my first full #ML course: Machine learning for Software Engineers from Educative, Inc. ... Highly recommend!"

Evan Dunbar

ML Engineer

"You guys are the gold standard of crash-courses... Narrow enough that it doesn't need years of study or a full blown book to get the gist, but broad enough that an afternoon of Googling doesn't cut it."

Carlos Matias La Borde

Software Developer

"I spend my days and nights on Educative. It is indispensable. It is such a unique and reader-friendly site"

Souvik Kundu

Front-end Developer

"Your courses are simply awesome, the depth they go into and the breadth of coverage is so good that I don't have to refer to 10 different websites looking for interview topics and content."

Vinay Krishnaiah

Software Developer

"I've tried probably 5-7 different sites and Educative is easily the best. It perfectly blends explanation with interactivity"

Eric Downs

Musician/Entrepeneur

Hands-on Learning Powered by AI

See how Educative uses AI to make your learning more immersive than ever before.

Instant Code Feedback

Evaluate and debug your code with the click of a button. Get real-time feedback on test cases, including time and space complexity of your solutions.

AI-Powered Mock Interviews

Put your skills to the test in a simulated interview setting. Receive personalized feedback based on your performance. Available in Premium & Premium Plus plans.

Adaptive Learning

At various checkpoints throughout Educative courses, you will be prompted to take a quick assessment. Receive a condensed curriculum tailored to your strengths and skill gaps.

Explain with AI

Select any text within any Educative course, and get an instant explanation — without ever leaving your browser.

AI Code Mentor

AI Code Mentor helps you quickly identify errors in your code, learn from your mistakes, and nudge you in the right direction — just like a 1:1 tutor!

Recommended before starting this course

Course

Learn to Code: Python for Absolute Beginners

Want to learn how to code? Get hands-on with Python for beginners. Master the basics, solve real-world problems, and confidently build in-demand Python skills for diverse industries.

beginner

8 hours

Course

Python 101: Interactively learn how to program with Python 3

This is a 101-level course in Python that will help you learn to program in Python for free.

beginner

10 hours

Course

Python Data Analysis and Visualization

Delve into Python data analysis and visualization using NumPy, SciPy, Pandas, and Seaborn. Gain insights into reading, cleaning, and visualizing data to generate actionable insights.

beginner

2 hours 30 mins

Course

Learn to Code: Python for Absolute Beginners

beginner

8 hour

Course

Python 101: Interactively learn how to program with Python 3

beginner

10 hour

Course

Python Data Analysis and Visualization

beginner

2 hour 30 min

Related Courses and Skill Paths

Course

Interactive Dashboards and Data Apps with Plotly and Dash

Gain insights into creating interactive dashboards and data apps with Plotly Dash. Learn to build, customize, and deploy data-driven applications with various controls and charts effectively.

beginner

27 hours

Course

Using Python Altair for Data Storytelling

Gain insights into using Python Altair for data storytelling. Learn about the DIKW pyramid and craft interactive visualizations to effectively convey complex ideas and narratives through data.

intermediate

17 hours

Course

Data Storytelling through Visualizations in Python

Gain insights into data storytelling in Python using Matplotlib, Seaborn, and Plotly. Explore trends, handle data challenges, and create impactful visual narratives aligning with business goals.

intermediate

9 hours

Course

Interactive Dashboards and Data Apps with Plotly and Dash

beginner

27 hour

Course

Using Python Altair for Data Storytelling

intermediate

17 hour

Course

Data Storytelling through Visualizations in Python