A crash course on making ML software more reliable, including best practices on testing and other aspects of defensive programming.

MyAwesomeDocker.tar.gz

MyAwesomeDocker

TorchJob

spa_run

spa_run_python

mypy_example

Ensuring the reliability and robustness of machine learning models is essential to building successful ML-powered applications.

This course begins with a thorough introduction to software testing essentials, particularly use cases within the machine learning context. You’ll learn about topics related to software testing, including unit and integration testing and more advanced testing techniques. Next, you’ll learn the best practices in software testing and dive into ML-specific testing techniques, such as behavioral and smoke tests. Lastly, you’ll cover the aspects of ML software reliability outside of testing, including runtime checks and type hinting.

By the end of this course, you'll be equipped with the knowledge and skills to ensure the reliability and robustness of your machine learning systems. You’ll be able to apply software engineering principles to your ML processes, create and execute efficient testing approaches, and utilize monitoring tools to identify and resolve problems in your ML systems.

Reliable Machine Learning

## Software testing

* 🛠 [pytest official documentation](https://docs.pytest.org/en/7.2.x/contents.html)

* 📃 [About fixtures in pytest](https://docs.pytest.org/en/7.1.x/explanation/fixtures.html)

* 📃 [pytest fixtures](https://docs.pytest.org/en/6.2.x/fixture.html): explicit, modular, scalable—a complete guide to using fixtures with all their methods, use cases, examples, and best practices.

* 📃 [Official pytest documentation on xfail](https://docs.pytest.org/en/latest/contents.html)

* 📃 [List of pytest plugins](https://docs.pytest.org/en/latest/reference/plugin_list.html)

* 📃 [XFail: mark test functions as expected to fail](https://docs.pytest.org/en/6.2.x/skipping.html#xfail-mark-test-functions-as-expected-to-fail)

* 🛠 [Hypothesis](https://hypothesis.readthedocs.io/en/latest/) is a Python library for property-based testing.

## Best and worst practices

- 📚 Steve McConnell, _Code Complete_.

### Test-driven development
- 📚 Kent Beck, _Test-Driven Development by Example_.

- 📃 ["Extreme Programming"](https://en.wikipedia.org/wiki/Extreme_programming)

* 📚 _Refactoring: Improving the Design of Existing Code_ by Martin Fowler, with Kent Beck

* 📃 [Test-driven development](https://en.wikipedia.org/wiki/Test-driven_development)


### Code coverage

* 🛠 [pytest-cov documentation](http://pytest-cov.rtfd.org/)

* 🛠 [pytest-cov multiprocessing](https://pytest-cov.readthedocs.io/en/latest/subprocess-support.html)

* 📃 [Code coverage](https://en.wikipedia.org/wiki/Code_coverage)

### CI/CD

* 🛠 [GitLab CI/CD](https://docs.gitlab.com/ee/ci/pipelines/)

* 🛠 [GitHub Actions](https://github.blog/2022-02-02-build-ci-cd-pipeline-github-actions-four-steps/)

* 🛠 [Drone CI](https://docs.drone.io/pipeline/overview/)

* 🛠 [CircleCI](https://circleci.com/)

## ML software readability

- 🛠 [Beyond Accuracy: Behavioral Testing of NLP Models with CheckList](https://homes.cs.washington.edu/~marcotcr/acl20_checklist.pdf)

### Runtime checks

* 📃 [Fail-safe](https://en.wikipedia.org/wiki/Fail-safe) 

* 📃 [Fail-fast](https://en.wikipedia.org/wiki/Fail-fast)  – Analogy: “Fail fast, fail often” is a famous mantra in Silicon Valley.

* 📃 [Failing badly](https://en.wikipedia.org/wiki/Failing_badly) vs Failing well

* 📃 [Fault-tolerant design](https://en.wikipedia.org/wiki/Fault_tolerance)

### Type hinting

* 🛠 [Python's type hinting](https://docs.python.org/3/library/typing.html)

* 📃 [PEP 483: A theory of type hints](https://peps.python.org/pep-0483/)

* 🛠 [MyPy: a static type checker for Python.](https://github.com/python/mypy) 

### Logging and debugging

* 🛠 [ELK stack: Elasticsearch, Kibana, and Logstash](https://www.elastic.co/what-is/elk-stack)

### Monitoring

* 🛠 [Prometheus](https://prometheus.io/): a tool for capturing metrics (monitoring and alerts).

* 🛠 [Grafana](https://grafana.com/): a tool for metrics visualization, analytics, etc.

* 🛠 [EvidentlyAI](https://evidentlyai.com/): A ML monitoring framework.

* 🛠 [NannyML](https://www.nannyml.com/): A library for post-deployment data science.

# Software testing

* 🛠 [pytest official documentation](https://docs.pytest.org/en/7.2.x/contents.html)

* 📃 [About fixtures in pytest](https://docs.pytest.org/en/7.1.x/explanation/fixtures.html)

* 📃 [pytest fixtures](https://docs.pytest.org/en/6.2.x/fixture.html): explicit, modular, scalable—a complete guide to using fixtures with all their methods, use cases, examples, and best practices.

* 📃 [Official pytest documentation on xfail](https://docs.pytest.org/en/latest/contents.html)

* 📃 [List of pytest plugins](https://docs.pytest.org/en/latest/reference/plugin_list.html)

* 📃 [XFail: mark test functions as expected to fail](https://docs.pytest.org/en/6.2.x/skipping.html#xfail-mark-test-functions-as-expected-to-fail)

* 🛠 [Hypothesis](https://hypothesis.readthedocs.io/en/latest/) is a Python library for property-based testing.

# Best and worst practices

- 📚 Steve McConnell, _Code Complete_.

## Test-driven development
- 📚 Kent Beck, _Test-Driven Development by Example_.

- 📃 ["Extreme Programming"](https://en.wikipedia.org/wiki/Extreme_programming)

* 📚 _Refactoring: Improving the Design of Existing Code_ by Martin Fowler, with Kent Beck

* 📃 [Test-driven development](https://en.wikipedia.org/wiki/Test-driven_development)


## Code coverage

* 🛠 [pytest-cov documentation](http://pytest-cov.rtfd.org/)

* 🛠 [pytest-cov multiprocessing](https://pytest-cov.readthedocs.io/en/latest/subprocess-support.html)

* 📃 [Code coverage](https://en.wikipedia.org/wiki/Code_coverage)

## CI/CD

* 🛠 [GitLab CI/CD](https://docs.gitlab.com/ee/ci/pipelines/)

* 🛠 [GitHub Actions](https://github.blog/2022-02-02-build-ci-cd-pipeline-github-actions-four-steps/)

* 🛠 [Drone CI](https://docs.drone.io/pipeline/overview/)

* 🛠 [CircleCI](https://circleci.com/)

# ML software readability

- 🛠 [Beyond Accuracy: Behavioral Testing of NLP Models with CheckList](https://homes.cs.washington.edu/~marcotcr/acl20_checklist.pdf)

## Runtime checks

* 📃 [Fail-safe](https://en.wikipedia.org/wiki/Fail-safe) 

* 📃 [Fail-fast](https://en.wikipedia.org/wiki/Fail-fast)  – Analogy: “Fail fast, fail often” is a famous mantra in Silicon Valley.

* 📃 [Failing badly](https://en.wikipedia.org/wiki/Failing_badly) vs Failing well

* 📃 [Fault-tolerant design](https://en.wikipedia.org/wiki/Fault_tolerance)

## Type hinting

* 🛠 [Python's type hinting](https://docs.python.org/3/library/typing.html)

* 📃 [PEP 483: A theory of type hints](https://peps.python.org/pep-0483/)

* 🛠 [MyPy: a static type checker for Python.](https://github.com/python/mypy) 

## Logging and debugging

* 🛠 [ELK stack: Elasticsearch, Kibana, and Logstash](https://www.elastic.co/what-is/elk-stack)

## Monitoring

* 🛠 [Prometheus](https://prometheus.io/): a tool for capturing metrics (monitoring and alerts).

* 🛠 [Grafana](https://grafana.com/): a tool for metrics visualization, analytics, etc.

* 🛠 [EvidentlyAI](https://evidentlyai.com/): A ML monitoring framework.

* 🛠 [NannyML](https://www.nannyml.com/): A library for post-deployment data science.

Read and watch additional info on reliable machine learning.

Introduction to Reliable ML

Software Testing

Best and Worst Practices

ML-Specific Tests

ML Software Reliability outside of Tests

Wrapping Up

Appendix

Links for Additional Reading

Software testing

Best and worst practices

Test-driven development

Code coverage

CI/CD

ML software readability

Runtime checks

Type hinting

Logging and debugging

Monitoring