Explore how to ensure reliability in ML models. Gain insights into software testing, ML-specific techniques, runtime checks, and monitoring tools to build robust ML systems effectively.

MyAwesomeDocker.tar.gz

MyAwesomeDocker

TorchJob

spa_run

spa_run_python

mypy_example

Ensuring the reliability and robustness of machine learning models is essential to building successful ML-powered applications.

This course begins with a thorough introduction to software testing essentials, particularly use cases within the machine learning context. You’ll learn about topics related to software testing, including unit and integration testing and more advanced testing techniques. Next, you’ll learn the best practices in software testing and dive into ML-specific testing techniques, such as behavioral and smoke tests. Lastly, you’ll cover the aspects of ML software reliability outside of testing, including runtime checks and type hinting.

By the end of this course, you'll be equipped with the knowledge and skills to ensure the reliability and robustness of your machine learning systems. You’ll be able to apply software engineering principles to your ML processes, create and execute efficient testing approaches, and utilize monitoring tools to identify and resolve problems in your ML systems.

Reliable Machine Learning

# Overview

Let's suppose our model runs on mobile and in the cloud using different engines—`coreml` and `onnxruntime`. It would be great to make sure both pipelines produce the same result for identical input, as shown below:

def test_pipelines_match(image_fixture):
    image = cv2.imread(image_fixture)
    coreml_model = CoreMLModel()
    onnx_model = OnnxModel()
    coreml_bbox, coreml_angle = coreml_model.predict(image)
    onnx_bbox, onnx_angle = onnx_model.predict(image)

    np.testing.assert_allclose(coreml_bbox, onnx_bbox, rtol=0, atol=1e-4)
    np.testing.assert_allclose(coreml_angle, onnx_angle, rtol=0, atol=1e-4)

# Consistency between `test` and `prod` environments

Let's look at the problem of development environments from a different angle. There is a common practice of using multiple stands for other purposes (and various stages of testing and production). We'll focus on two main stands:

* **`test` stand:** Also known as the **testing environment**, this is a particular environment where we check if everything works, before showing our updates to the stakeholders and/or users. We could also use this stand for things like autotests and load testing.

* **`prod` stand:** Also known as the **production environment**, this environment, including the code, models, data, etc., works with real user traffic. 


Our ML model's behavior should be consistent between the `test` and `prod` stands and give the same output for the same data. Sometimes, it's not possible even with a fixated `random_state` (for example, when dealing with neural networks, in this case, we should at least get almost the same accuracy in both environments). But it's better to bring the testing and production environments as close as possible.


Learn how to ensure that the testing and production environments are equal to catch problems.

Introduction to Reliable ML

Software Testing

Best and Worst Practices

ML-Specific Tests

ML Software Reliability outside of Tests

Wrapping Up

Appendix

Comparing the Environments

Overview

Consistency between `test` and `prod` environments

Comparing the Environments

Overview

Consistency between test and prod environments

Consistency between `test` and `prod` environments