A crash course on making ML software more reliable, including best practices on testing and other aspects of defensive programming.

MyAwesomeDocker.tar.gz

MyAwesomeDocker

TorchJob

spa_run

spa_run_python

mypy_example

Ensuring the reliability and robustness of machine learning models is essential to building successful ML-powered applications.

This course begins with a thorough introduction to software testing essentials, particularly use cases within the machine learning context. You’ll learn about topics related to software testing, including unit and integration testing and more advanced testing techniques. Next, you’ll learn the best practices in software testing and dive into ML-specific testing techniques, such as behavioral and smoke tests. Lastly, you’ll cover the aspects of ML software reliability outside of testing, including runtime checks and type hinting.

By the end of this course, you'll be equipped with the knowledge and skills to ensure the reliability and robustness of your machine learning systems. You’ll be able to apply software engineering principles to your ML processes, create and execute efficient testing approaches, and utilize monitoring tools to identify and resolve problems in your ML systems.

Reliable Machine Learning

**(Select all that apply.)** Which test types are behavioral tests?

(Select all that apply.) Which test types are behavioral tests?

The model is tested under the same conditions in which it will work.

The end user of each environment is the same.

The data in both environments is the same.

The network in which both environments are operating should be the same.

**(Select all that apply.)** What should we pay attention to when comparing our testing and production environments?

(Select all that apply.) What should we pay attention to when comparing our testing and production environments?

The quality of the model not changing over time

Unexpected updates in a library used by our model

The model's performance is consistent for different random seeds.

**(Select all that apply.)** What can tests for exact values prevent?

(Select all that apply.) What can tests for exact values prevent?

Testing if the neural network’s layers match each other.

Testing if the CI pipeline at least compiles.

Testing if the model class at least imports.

Testing if the data pipeline at least compiles.

Testing if the ML model can overfit a single batch of the data.

Testing if the model at least predicts something on 1,000,000 samples.

**(Select all that apply.)** Where can smoke tests be helpful?

(Select all that apply.) Where can smoke tests be helpful?

To significantly reduce the time required to run a testable model training loop

A lightweight imitation of the environment in which the ML model will run

Using the once-fitted model in different tests

Use once created dataset in different tests

To test the quality of the model on the small dataset

**(Select all that apply.)** What are the reasons for using fixtures and mocking in machine learning?

(Select all that apply.) What are the reasons for using fixtures and mocking in machine learning?

Take a short quiz to test how well you understand machine learning code testing.

Introduction to Reliable ML

Software Testing

Best and Worst Practices

ML-Specific Tests

ML Software Reliability outside of Tests

Wrapping Up

Appendix

Quiz: ML-Specific Tests