Explore how to ensure reliability in ML models. Gain insights into software testing, ML-specific techniques, runtime checks, and monitoring tools to build robust ML systems effectively.

MyAwesomeDocker.tar.gz

MyAwesomeDocker

TorchJob

spa_run

spa_run_python

mypy_example

Ensuring the reliability and robustness of machine learning models is essential to building successful ML-powered applications.

This course begins with a thorough introduction to software testing essentials, particularly use cases within the machine learning context. You’ll learn about topics related to software testing, including unit and integration testing and more advanced testing techniques. Next, you’ll learn the best practices in software testing and dive into ML-specific testing techniques, such as behavioral and smoke tests. Lastly, you’ll cover the aspects of ML software reliability outside of testing, including runtime checks and type hinting.

By the end of this course, you'll be equipped with the knowledge and skills to ensure the reliability and robustness of your machine learning systems. You’ll be able to apply software engineering principles to your ML processes, create and execute efficient testing approaches, and utilize monitoring tools to identify and resolve problems in your ML systems.

Reliable Machine Learning

# Overview

**Smoke tests** check if the code works at all and whether it's testable in the first place. Smoke testing verifies the most basic assumptions, like a doctor verifying whether the patient is alive (do they have a pulse) before analyzing the state of their health.

Smoke tests scratch the surface of our piece of software. They are very high-level integration tests. Some people prefer calling them sanity checks. For script languages like Python, such tests can be compared to something as simple as "_it compiles_."

> "The phrase smoke test comes from electronic hardware testing. You plug in a new board and turn on the power. If you see smoke coming from the board, turn off the power. You don't have to do any more testing."—Cem Kaner, James Bach, Brett Pettichord, _Lessons Learned in Software Testing_.

# When to use smoke tests?

The concept of a smoke test is useful when an engineer needs to start working with a complex codebase that is unknown to them and hardly covered by tests. Working with such code may initially sound complicated, but it’s doable.

To give you an intuitive sense of smoke tests, here are a couple of examples:

* **Check if all required imports work.** This will test whether the paths are set correctly and whether modules are visible for the Python interpreter. Also, it ensures there is no crucial typo in our Python files and they are interpreted correctly.

* **Check if `config` is found and contains the required fields.** This will prevent anyone from making unintentional changes to its structure.

* **Run some complex functions and ensure they return something.** Through these tests, you can check if they don't raise an error. Moreover, you can verify if the output format matches your expectations.

# Testing end-to-end pipeline

When working with different pipelines, it's a common practice to ensure the whole pipeline works by running it on some sample of data. Let's look at some sample pipelines below.

## Deep learning

In the deep learning world, we check if all variables are set correctly, the loss decreases over time, and it can overfit a tiny toy dataset (for example, on a single batch). 



Learn the fastest way to check if the ML/DL/ETL pipeline works.

Introduction to Reliable ML

Software Testing

Best and Worst Practices

ML-Specific Tests

ML Software Reliability outside of Tests

Wrapping Up

Appendix

Smoke Tests for the ML Pipeline

Overview

When to use smoke tests?