Search⌘ K
Home
Courses
Reliable Machine Learning

Comparing the Environments

Explore how to ensure consistency between testing and production environments in machine learning. Understand the risks of environmental differences and learn best practices to align test and production settings to maintain model accuracy and pipeline reliability.

We'll cover the following...

Overview

Let’s suppose our model runs on mobile and in the cloud using different engines—coreml and onnxruntime. It would be great to make sure both pipelines produce the same result for identical input, as shown below:

Python 3.8
def test_pipelines_match(image_fixture):
    image = cv2.imread(image_fixture)
    coreml_model = CoreMLModel()
    onnx_model = OnnxModel()
    coreml_bbox, coreml_angle = coreml_model.predict(image)
    onnx_bbox, onnx_angle = onnx_model.predict(image)
    np.testing.assert_allclose(coreml_bbox, onnx_bbox, rtol=0, atol=1e-4)
    np.testing.assert_allclose(coreml_angle, onnx_angle, rtol=0, atol=1e-4)

Consistency between test and prod environments

Let’s look at the problem of development environments from a different angle. There is a common practice of using multiple stands for other purposes (and various stages of testing and production). We’ll focus on ...