Negative Testing: Relaxing Constraints

Relaxing constraints

While metrics are always a good thing to keep an eye on, another very interesting way to improve our negative tests and explore the program’s problem space is to play with constraints and relax them with existing generators.

The way we relax constraints is usually through simple code modifications. We’ll modify a bunch of generators, making them less strict so they trigger some unexpected cases. Then we find out why that happened, revert the change, and then either add a unit test or a property test to validate the bug before fixing it.

Then we can rinse and repeat, gradually weeding out more and more bugs from our code.

Let’s start with the first one, checking what happens when item names are not unique in the price list, which we can do by changing the last line of the price_list() generator like this:

%% generate a list of {ItemName, Price} to configure the checkout
price_list() ->
    ?LET(PriceList, non_empty(list({non_empty(string()), integer()})),
        lists:keysort(1, PriceList)). % allow duplicates

Note: Replace the price_list generator in the code below to see the output.

Here we replaced lists:ukeysort/2 with lists:keysort/2, preserving similar ...