Flaky and Negative Tests

Negative tests

Negative tests, also known as exception tests, are essential to any machine learning project. They ensure the code can handle incorrect input or unexpected scenarios and raises an error appropriately. This is important because it helps to identify and prevent potential issues that could lead to incorrect or unexpected results.

When positive tests check that everything is good (optimistic scenario), negative tests check various (mostly graceful) potential failures. The more experienced the engineer is, the more negative tests they usually build.

“Developer tests tend to be ‘clean tests.’ Developers tend to test whether the code works (clean tests), rather than all the ways the code breaks (dirty tests). Immature testing organizations have about five clean tests for every dirty test. Mature testing organizations tend to have five dirty tests for every clean test. This ratio is not reversed by reducing the clean tests; it’s done by creating 25 times as many dirty tests (Boris Beizer in Johnson 1994).”

—Steve McConnell, Code Complete.

Why are negative tests important?

The fail fast paradigm states that it’s better to identify and fix errors as soon as possible rather than letting them go unnoticed and potentially causing issues down the line. This is particularly important in machine learning projects where errors can be difficult to detect and fix.

The GIGO (garbage in–garbage out) principle states that if the input data is flawed or incorrect, the output will also be flawed. Therefore, negative tests can help ensure that the input data is valid and accurate, which will lead to more reliable and accurate results.

Additionally, by catching errors early on, it becomes easier to localize the source of the issue and fix it. Overall, negative tests play a crucial role in ensuring the reliability of machine learning projects by helping to identify and fix errors early on.