Explore how to ensure reliability in ML models. Gain insights into software testing, ML-specific techniques, runtime checks, and monitoring tools to build robust ML systems effectively.

MyAwesomeDocker.tar.gz

MyAwesomeDocker

TorchJob

spa_run

spa_run_python

mypy_example

Ensuring the reliability and robustness of machine learning models is essential to building successful ML-powered applications.

This course begins with a thorough introduction to software testing essentials, particularly use cases within the machine learning context. You’ll learn about topics related to software testing, including unit and integration testing and more advanced testing techniques. Next, you’ll learn the best practices in software testing and dive into ML-specific testing techniques, such as behavioral and smoke tests. Lastly, you’ll cover the aspects of ML software reliability outside of testing, including runtime checks and type hinting.

By the end of this course, you'll be equipped with the knowledge and skills to ensure the reliability and robustness of your machine learning systems. You’ll be able to apply software engineering principles to your ML processes, create and execute efficient testing approaches, and utilize monitoring tools to identify and resolve problems in your ML systems.

Reliable Machine Learning

## Overview

Tests are considered reasonable if they frequently run, preferably each time we commit to our feature branch. Tests should not slow down the coding process of a programmer, so their design should be as minimal as possible: no complexity and computation, except minimal viable ones. Otherwise, the developer is tempted to disable their tests if they take more time than expected.

Here, lightweight fixtures come to the rescue. They can be patched instead of real models or data to speed up testing drastically. We refer to tests with these kinds of insertions as **mock tests**.

## Shallow network as a fixture
 
Instead of using a large network, we can have a dummy replacement with similar properties (e.g., input and output dimensions), so we could use it in integration tests while keeping them fast enough.

Let's look at the example below where a dummy `Resnet50` backbone takes a 224x224x3 image as input and returns a 1x2048 feature vector as output.



# Overview

Tests are considered reasonable if they frequently run, preferably each time we commit to our feature branch. Tests should not slow down the coding process of a programmer, so their design should be as minimal as possible: no complexity and computation, except minimal viable ones. Otherwise, the developer is tempted to disable their tests if they take more time than expected.

Here, lightweight fixtures come to the rescue. They can be patched instead of real models or data to speed up testing drastically. We refer to tests with these kinds of insertions as **mock tests**.

# Shallow network as a fixture
 
Instead of using a large network, we can have a dummy replacement with similar properties (e.g., input and output dimensions), so we could use it in integration tests while keeping them fast enough.

Let's look at the example below where a dummy `Resnet50` backbone takes a 224x224x3 image as input and returns a 1x2048 feature vector as output.



Learn how to use fixtures and mocks in machine learning tests to save time during test runs.

Introduction to Reliable ML

Software Testing

Best and Worst Practices

ML-Specific Tests

ML Software Reliability outside of Tests

Wrapping Up

Appendix

Fixtures and Mocks in Machine Learning Tests

Overview

Shallow network as a fixture