Reliable Machine Learning/

...

/

Comparing the Environments

Comparing the Environments

Learn how to ensure that the testing and production environments are equal to catch problems.

We'll cover the following...

Overview

Let’s suppose our model runs on mobile and in the cloud using different engines—coreml and onnxruntime. It would be great to make sure both pipelines produce the same result for identical input, as shown below:

def test_pipelines_match(image_fixture):
    image = cv2.imread(image_fixture)
    coreml_model = CoreMLModel()
    onnx_model = OnnxModel()
    coreml_bbox, coreml_angle = coreml_model.predict(image)
    onnx_bbox, onnx_angle = onnx_model.predict(image)
    np.testing.assert_allclose(coreml_bbox, onnx_bbox, rtol=0, atol=1e-4)
    np.testing.assert_allclose(coreml_angle, onnx_angle, rtol=0, atol=1e-4)

Consistency between test and prod environments

Let’s look at the problem of development environments from a different angle. There is a common practice of using multiple stands for other purposes (and various stages of testing and production). We’ll focus on ...