Explore how to ensure reliability in ML models. Gain insights into software testing, ML-specific techniques, runtime checks, and monitoring tools to build robust ML systems effectively.

MyAwesomeDocker.tar.gz

MyAwesomeDocker

TorchJob

spa_run

spa_run_python

mypy_example

Ensuring the reliability and robustness of machine learning models is essential to building successful ML-powered applications.

This course begins with a thorough introduction to software testing essentials, particularly use cases within the machine learning context. You’ll learn about topics related to software testing, including unit and integration testing and more advanced testing techniques. Next, you’ll learn the best practices in software testing and dive into ML-specific testing techniques, such as behavioral and smoke tests. Lastly, you’ll cover the aspects of ML software reliability outside of testing, including runtime checks and type hinting.

By the end of this course, you'll be equipped with the knowledge and skills to ensure the reliability and robustness of your machine learning systems. You’ll be able to apply software engineering principles to your ML processes, create and execute efficient testing approaches, and utilize monitoring tools to identify and resolve problems in your ML systems.

Reliable Machine Learning

We should prepare test samples before fitting the model.

We should always write tests after we write the code.

We should add a new test each time a new bug occurs.

What does test-driven development mean?

Measure the percentage of code covered with tests

Highlight the areas of the code not covered with the tests yet

Automatically measures if our code is tested enough

Measure the quality (i.e., depth) of the written tests

**(Select all that apply.)** What does code coverage do?

(Select all that apply.) What does code coverage do?

Ask each member in our team to run tests before the merge request.

Put running tests into our project's schedule (e.g., on a weekly basis).

Make tests a part of our code so that they, as a consequence, run regularly.

How do we test our code on a regular basis?

How do we test our code on a regular basis?

How our NLP model reacts to negative words

How our neural network reacts to wrong shape data as an input

How our classifier reacts to negative samples

**(Select all that apply.)** What do negative tests check?

(Select all that apply.) What do negative tests check?

Incorrect design of IO operations with temporary data.

Here, we run some deterministic queries to the database. Why is this test flaky?

```python
def test_ch_query(query, df_expected):
    from clickhouse_driver import Client

    config = parse_config("config.ini")
    client = Client(**config)

    df = client.query_frame(query)
    pd.assert_frame_equal(df, df_excected)
```

The content of the `config.ini` file:
```
[clickhouse.connection]
clickhouse-host = clickhouse.lab.reliable.ml
clickhouse-user = xaea12
clickhouse-pass = querty
```

Here, we run some deterministic queries to the database. Why is this test flaky?
<pre><code class="language-python">def test_ch_query(query, df_expected):
 from clickhouse_driver import Client

 config = parse_config(&quot;config.ini&quot;)
 client = Client(**config)

 df = client.query_frame(query)
 pd.assert_frame_equal(df, df_excected)
</code></pre>
The content of the <code>config.ini</code> file:
<pre><code>[clickhouse.connection]
clickhouse-host = clickhouse.lab.reliable.ml
clickhouse-user = xaea12
clickhouse-pass = querty
</code></pre>

Tests marked with `xfail` but returned both `x` and `X` marks.

Tests that operate the stuff in parallel.

**(Select all that apply.)** Which tests are called flaky?

(Select all that apply.) Which tests are called flaky?

Take a short quiz to test how well you understand the best and worse practices in software testing.

Introduction to Reliable ML

Software Testing

Best and Worst Practices

ML-Specific Tests

ML Software Reliability outside of Tests

Wrapping Up

Appendix

Quiz: Best and Worst Practices