A crash course on making ML software more reliable, including best practices on testing and other aspects of defensive programming.

MyAwesomeDocker.tar.gz

MyAwesomeDocker

TorchJob

spa_run

spa_run_python

mypy_example

Ensuring the reliability and robustness of machine learning models is essential to building successful ML-powered applications.

This course begins with a thorough introduction to software testing essentials, particularly use cases within the machine learning context. You’ll learn about topics related to software testing, including unit and integration testing and more advanced testing techniques. Next, you’ll learn the best practices in software testing and dive into ML-specific testing techniques, such as behavioral and smoke tests. Lastly, you’ll cover the aspects of ML software reliability outside of testing, including runtime checks and type hinting.

By the end of this course, you'll be equipped with the knowledge and skills to ensure the reliability and robustness of your machine learning systems. You’ll be able to apply software engineering principles to your ML processes, create and execute efficient testing approaches, and utilize monitoring tools to identify and resolve problems in your ML systems.

Reliable Machine Learning

An Instagram filter is not sure where the face is on a selfie.

A self-driving car in a GTA simulation ran over a man.

TikTok’s recommendation could not pick up a video in which it is `>95 percent` confident.

**(Select all that apply.)** Which option should our ML service return no result (or fallback response) to?

(Select all that apply.) Which option should our ML service return no result (or fallback response) to?

No, Python is a dynamic programming language, so we can change the type of the variable on the fly.

Yes, it improves code understanding for other engineers.

Yes, if we have enough time; otherwise, we should skip it.

Do we need type hinting in Python?

Yes, we should keep only the essential one and drop the rarely used logging.

Things may vary, depending on the project.

**(Select all that apply.)** What can data quality monitoring be used for?

(Select all that apply.) What can data quality monitoring be used for?

Always provide the model with samples from the same distribution.

Detect distribution changes in the dataset used for training.

Measure service health on a regular basis (e.g., daily or hourly).

**(Select all that apply.)** How do we prevent model drift?

(Select all that apply.) How do we prevent model drift?

Take a short quiz to test how well you understand machine learning software reliability.

Introduction to Reliable ML

Software Testing

Best and Worst Practices

ML-Specific Tests

ML Software Reliability outside of Tests

Wrapping Up

Appendix

Quiz: ML Software Reliability Outside of Tests