Explore how to ensure reliability in ML models. Gain insights into software testing, ML-specific techniques, runtime checks, and monitoring tools to build robust ML systems effectively.

MyAwesomeDocker.tar.gz

MyAwesomeDocker

TorchJob

spa_run

spa_run_python

mypy_example

Ensuring the reliability and robustness of machine learning models is essential to building successful ML-powered applications.

This course begins with a thorough introduction to software testing essentials, particularly use cases within the machine learning context. You’ll learn about topics related to software testing, including unit and integration testing and more advanced testing techniques. Next, you’ll learn the best practices in software testing and dive into ML-specific testing techniques, such as behavioral and smoke tests. Lastly, you’ll cover the aspects of ML software reliability outside of testing, including runtime checks and type hinting.

By the end of this course, you'll be equipped with the knowledge and skills to ensure the reliability and robustness of your machine learning systems. You’ll be able to apply software engineering principles to your ML processes, create and execute efficient testing approaches, and utilize monitoring tools to identify and resolve problems in your ML systems.

Reliable Machine Learning

# ML-specific software

In regular software,  bugs tend to be more explicit. In ML software, defects are much easier to hide. Let’s consider the code snippet below, where we have a simple function from a computer vision pipeline.



def extract_angle_from_warp_matrix(rotation_matrix: np.ndarray) -> np.float:
    # Extracts rotation angle (in degrees) from warp matrix
    return np.rad2deg(np.arctan2(-rotation_matrix[1, 0], rotation_matrix[1, 1]))

Even code authors (people who probably understand the domain) might have a hard time understanding if this snippet is correct. Do we need the `-` sign next to `rotation_matrix[1, 0]`? What’s the structure of this array? We might find it difficult to remember without additional references.

For regular software, things are more straightforward, such that running the code or carefully reading the code often reveals problems. However, defects in ML-related software can go unnoticed and cause problems for the final model performance. 


# Try it yourself

Try to find out if there is a bug in the code below. If there is, try to fix the bug using the hints provided.




Learn how machine learning is different from regular software.

Introduction to Reliable ML

Software Testing

Best and Worst Practices

ML-Specific Tests

ML Software Reliability outside of Tests

Wrapping Up

Appendix

Why Machine Learning is Specific

ML-specific software