Explore how to ensure reliability in ML models. Gain insights into software testing, ML-specific techniques, runtime checks, and monitoring tools to build robust ML systems effectively.

MyAwesomeDocker.tar.gz

MyAwesomeDocker

TorchJob

spa_run

spa_run_python

mypy_example

Ensuring the reliability and robustness of machine learning models is essential to building successful ML-powered applications.

This course begins with a thorough introduction to software testing essentials, particularly use cases within the machine learning context. You’ll learn about topics related to software testing, including unit and integration testing and more advanced testing techniques. Next, you’ll learn the best practices in software testing and dive into ML-specific testing techniques, such as behavioral and smoke tests. Lastly, you’ll cover the aspects of ML software reliability outside of testing, including runtime checks and type hinting.

By the end of this course, you'll be equipped with the knowledge and skills to ensure the reliability and robustness of your machine learning systems. You’ll be able to apply software engineering principles to your ML processes, create and execute efficient testing approaches, and utilize monitoring tools to identify and resolve problems in your ML systems.

Reliable Machine Learning

# Negative tests

Negative tests, also known as exception tests, are essential to any machine learning project. They ensure the code can handle incorrect input or unexpected scenarios and raises an error appropriately. This is important because it helps to identify and prevent potential issues that could lead to incorrect or unexpected results.
 
When positive tests check that everything is good (optimistic scenario), negative tests check various (mostly graceful) potential failures. The more experienced the engineer is, the more negative tests they usually build.

> "Developer tests tend to be 'clean tests.' Developers tend to test whether the code works (clean tests), rather than all the ways the code breaks (dirty tests). Immature testing organizations have about five clean tests for every dirty test. Mature testing organizations tend to have five dirty tests for every clean test. This ratio is not reversed by reducing the clean tests; it's done by creating 25 times as many dirty tests (Boris Beizer in Johnson 1994)."
> 
>—Steve McConnell, _Code Complete_.

## Why are negative tests important?

The **fail fast paradigm** states that it's better to identify and fix errors as soon as possible rather than letting them go unnoticed and potentially causing issues down the line. This is particularly important in machine learning projects where errors can be difficult to detect and fix. 

The **GIGO (garbage in–garbage out) principle** states that if the input data is flawed or incorrect, the output will also be flawed. Therefore, negative tests can help ensure that the input data is valid and accurate, which will lead to more reliable and accurate results.

Additionally, by catching errors early on, it becomes easier to localize the source of the issue and fix it. Overall, negative tests play a crucial role in ensuring the reliability of machine learning projects by helping to identify and fix errors early on.

Learn about the importance of flaky and negative tests for ensuring the reliability of machine learning projects.

Introduction to Reliable ML

Software Testing

Best and Worst Practices

ML-Specific Tests

ML Software Reliability outside of Tests

Wrapping Up

Appendix

Flaky and Negative Tests

Negative tests