Explore how to ensure reliability in ML models. Gain insights into software testing, ML-specific techniques, runtime checks, and monitoring tools to build robust ML systems effectively.

MyAwesomeDocker.tar.gz

MyAwesomeDocker

TorchJob

spa_run

spa_run_python

mypy_example

Ensuring the reliability and robustness of machine learning models is essential to building successful ML-powered applications.

This course begins with a thorough introduction to software testing essentials, particularly use cases within the machine learning context. You’ll learn about topics related to software testing, including unit and integration testing and more advanced testing techniques. Next, you’ll learn the best practices in software testing and dive into ML-specific testing techniques, such as behavioral and smoke tests. Lastly, you’ll cover the aspects of ML software reliability outside of testing, including runtime checks and type hinting.

By the end of this course, you'll be equipped with the knowledge and skills to ensure the reliability and robustness of your machine learning systems. You’ll be able to apply software engineering principles to your ML processes, create and execute efficient testing approaches, and utilize monitoring tools to identify and resolve problems in your ML systems.

Reliable Machine Learning

# Overview

When the code of our machine learning service is written and put into production, it should run as smoothly as a living organism. We don't always have the option to intervene and fix something on the go. There aren't many things to do to make this code reliable, but we'll discuss most of them in this section.

## Fail fast vs. fail safe

Depending on the domain and application, we should either process exceptions/bugs silently (log or trigger an alert and continue running) or completely stop runtime. This decision is also called the robustness vs. correctness issue.



# Robustness vs. correctness

Steve McConnell, in _Code Complete_ precisely highlights the difference between robustness and correctness: 

> "As the video game and x-ray examples show us, the most appropriate error processing style depends on the kind of software the error occurs in. These examples also illustrate that error processing generally favors more correctness or robustness. Developers tend to use these terms informally, but, strictly speaking, these terms are at opposite ends of the scale from each other. Correctness means never returning an inaccurate result; returning no result is better than returning an inaccurate result. Robustness means always trying to do something that will allow the software to keep operating, even if that leads to results that are inaccurate sometimes.
>
> ...
>
> Safety-critical applications tend to favor correctness to robustness. It is better to return no result than to return a wrong result. The radiation machine is a good example of this principle.
> Consumer applications tend to favor robustness to correctness. Any result whatsoever is usually better than the software shutting down. The word processor I'm using occasionally displays a fraction of a line of text at the bottom of the screen. If it detects that condition, do I want the word processor to shut down?"

It's important to consider the trade-offs between robustness and correctness when designing and implementing any system, as well as the potential consequences of errors or failures. 

In safety-critical applications, it's usually more important to prioritize correctness over robustness. In contrast, it's often more important in consumer applications to prioritize robustness over correctness because users may be more forgiving of imperfections or errors as long as the overall system can continue functioning.

## Machine learning examples

This principle is very applicable to ML pipelines as well. Imagine we’re building an entertainment app like an AR mask for Snap or TikTok. In the worst-case scenario, the added effect will look unpleasant for a frame, which is not that big a risk, so robustness is a reasonable goal here.

On the contrary, consider an ML solution for medical or transportation needs. Would we like a self-driving car that moves forward when unsure if a pedestrian is nearby?



Learn about assertions and when to run them.

Introduction to Reliable ML

Software Testing

Best and Worst Practices

ML-Specific Tests

ML Software Reliability outside of Tests

Wrapping Up

Appendix

Runtime Checks

Overview

Fail fast vs. fail safe

Robustness vs. correctness