A crash course on making ML software more reliable, including best practices on testing and other aspects of defensive programming.

MyAwesomeDocker.tar.gz

MyAwesomeDocker

TorchJob

spa_run

spa_run_python

mypy_example

Ensuring the reliability and robustness of machine learning models is essential to building successful ML-powered applications.

This course begins with a thorough introduction to software testing essentials, particularly use cases within the machine learning context. You’ll learn about topics related to software testing, including unit and integration testing and more advanced testing techniques. Next, you’ll learn the best practices in software testing and dive into ML-specific testing techniques, such as behavioral and smoke tests. Lastly, you’ll cover the aspects of ML software reliability outside of testing, including runtime checks and type hinting.

By the end of this course, you'll be equipped with the knowledge and skills to ensure the reliability and robustness of your machine learning systems. You’ll be able to apply software engineering principles to your ML processes, create and execute efficient testing approaches, and utilize monitoring tools to identify and resolve problems in your ML systems.

Reliable Machine Learning

## Overview

The machine learning world is full of randomness, inaccuracy, and uncertainty. Often, we cannot test the exact values of something (like in regular deterministic software). Still, there are things or basic logic that remain unchanged in the behavior of our models and auxiliary functions for them.

**Behavioral tests** are a crucial part of the machine learning process because they help ensure our models are functioning as expected and producing reliable results. These tests examine the model and auxiliary functions under different conditions and input variations to identify potential issues or inconsistencies.

Behavioral testing involves various techniques and methods, such as consistency, invariance, negation, and directional expectation tests:

* **Consistency tests** involve checking whether different data transformations are consistent with each other.

* **Invariance tests** check whether particular properties of the model (called invariances) remain unchanged when given input that has been transformed or distorted in some way. 

* **Negation tests** involve checking whether the model produces the opposite output when given the opposite of the expected input. 

* **Directional expectation tests** involve checking whether the model changes its output in the expected direction when given input that deviates from the initial input in a specific way.

## Consistency tests

Consistency tests ensure that the behavior of functions related to each other is consistent. In other words, they check that any transformations supposed to invert each other do so.

For example, in data preprocessing, we often use functions to convert things back and forth (e.g., converting from radians to degrees and vice versa, `np.rad2deg` and `np.deg2rad`).

In the world of computer vision, popular conversions are: 
- different units (degrees to radians) 
- different representations (quaternions vs. rotation matrices). 
- different formats (`torch` vs. `numpy`, `uint` 0..255 vs. `float` 0..1).

### Consistency using unit tests

We may require consistency even in a simple unit test. 



# Overview

The machine learning world is full of randomness, inaccuracy, and uncertainty. Often, we cannot test the exact values of something (like in regular deterministic software). Still, there are things or basic logic that remain unchanged in the behavior of our models and auxiliary functions for them.

**Behavioral tests** are a crucial part of the machine learning process because they help ensure our models are functioning as expected and producing reliable results. These tests examine the model and auxiliary functions under different conditions and input variations to identify potential issues or inconsistencies.

Behavioral testing involves various techniques and methods, such as consistency, invariance, negation, and directional expectation tests:

* **Consistency tests** involve checking whether different data transformations are consistent with each other.

* **Invariance tests** check whether particular properties of the model (called invariances) remain unchanged when given input that has been transformed or distorted in some way. 

* **Negation tests** involve checking whether the model produces the opposite output when given the opposite of the expected input. 

* **Directional expectation tests** involve checking whether the model changes its output in the expected direction when given input that deviates from the initial input in a specific way.

# Consistency tests

Consistency tests ensure that the behavior of functions related to each other is consistent. In other words, they check that any transformations supposed to invert each other do so.

For example, in data preprocessing, we often use functions to convert things back and forth (e.g., converting from radians to degrees and vice versa, `np.rad2deg` and `np.deg2rad`).

In the world of computer vision, popular conversions are: 
- different units (degrees to radians) 
- different representations (quaternions vs. rotation matrices). 
- different formats (`torch` vs. `numpy`, `uint` 0..255 vs. `float` 0..1).

## Consistency using unit tests

We may require consistency even in a simple unit test. 



Learn about behavioral testing, including consistency, invariance, and directional expectation tests.

Introduction to Reliable ML

Software Testing

Best and Worst Practices

ML-Specific Tests

ML Software Reliability outside of Tests

Wrapping Up

Appendix

Behavioral Testing

Overview

Consistency tests

Consistency using unit tests