Explore how to ensure reliability in ML models. Gain insights into software testing, ML-specific techniques, runtime checks, and monitoring tools to build robust ML systems effectively.

MyAwesomeDocker.tar.gz

MyAwesomeDocker

TorchJob

spa_run

spa_run_python

mypy_example

Ensuring the reliability and robustness of machine learning models is essential to building successful ML-powered applications.

This course begins with a thorough introduction to software testing essentials, particularly use cases within the machine learning context. You’ll learn about topics related to software testing, including unit and integration testing and more advanced testing techniques. Next, you’ll learn the best practices in software testing and dive into ML-specific testing techniques, such as behavioral and smoke tests. Lastly, you’ll cover the aspects of ML software reliability outside of testing, including runtime checks and type hinting.

By the end of this course, you'll be equipped with the knowledge and skills to ensure the reliability and robustness of your machine learning systems. You’ll be able to apply software engineering principles to your ML processes, create and execute efficient testing approaches, and utilize monitoring tools to identify and resolve problems in your ML systems.

Reliable Machine Learning

# Overview

**Fixtures** are fixed environment items (usually function inputs and outputs) used in tests. Without fixtures, we do not separate test-case preparation logic from the test itself. Moreover, we're probably creating almost the same test cases for different tests.

Let’s look at the code below where we add some images to our repository and use the `crop_image` function to crop an image.



from typing import Tuple
import numpy as np


def crop_image(img: np.ndarray, coords: Tuple[int, int, int, int]) -> np.ndarray:
    x, y = coords[0], coords[1]
    w, h = coords[2], coords[3]
    return img[x:x+w, y:y+h]

import cv2
import numpy as np
from utils import crop_image


def test_crop_image_with_fixture():
    full_img = cv2.imread('./fixtures/full_img.png')
    crop_img = cv2.imread('./fixtures/crop_img.png')
    cropped = crop_image(full_img, (10, 20, 40, 40))

    np.testing.assert_equal(cropped, crop_img)

import React from 'react';
import { shallow, configure } from 'enzyme';
import Adapter from 'enzyme-adapter-react-16';

configure({ adapter: new Adapter() });

import HelloWorld from './app';

var TestResult = function() {
    this.succeeded = false;
    this.reason = "";
    this.input = "";
    this.expected_output = "";
    this.actual_output = "";
}

export const executeTests = function() {

  var results = [];

  result = new TestResult();
  result.input = 'HelloWorld Component';
  result.expected_output = "span containing text 'Hello World'"

  let wrapper = shallow(<HelloWorld />);

  // Call your Challenge function here.

  let type = wrapper.type();
  let testSuccessful = true;
  let failureReason;

  if (type !== 'span') {
    testSuccessful = false;
    failureReason = "You need to render exactly one span HTML element";
  } else if (wrapper.props().children != "Hello World") {
     testSuccessful = false;
     failureReason = "You have rendered wrong message in your span element";
  }

  result.actual_output = wrapper.html();

  if (testSuccessful) {
    result.succeeded = true;
    result.reason = "Succeeded"
  } else {
    result.succeeded = false;
    result.reason = failureReason;
  }

  results.push(result);

  return results;
}


React

Dart

GoJS React

Typescript React

Vue.js

Another way to use fixtures is by passing them through the test's arguments (which can only be either the fixture or the test's parameter), while creating or loading fixtures separately by using the `@pytest.fixture` decorator.

Let’s demonstrate this on the same test that we used earlier.


Learn about fixtures and how to use them in your tests.

Introduction to Reliable ML

Software Testing

Best and Worst Practices

ML-Specific Tests

ML Software Reliability outside of Tests

Wrapping Up

Appendix

Fixtures in pytest

Overview