Filtering Records

Testing date

We have a module to convert CSV to maps, and we know that we’ll need employee records and a way to filter them to move the project forward. We could start by defining the records’ specific fields, but since we know they’ll be implemented using maps, and that maps are a fairly dynamic data structure, nothing prevents us from jumping directly to the filtering step.

Problem size

In this lesson, we’ll see a case where even though the problem space is large, we can explore it better with example tests than with properties. This is because the type of data we can get is very regular and easy to enumerate, such that a brute force strategy pretty much guarantees a more exhaustive and reliable testing approach than a probabilistic one with properties.

For example, by filtering employee maps based on dates, we’ll see that even when they’re not the best tool, properties can still be a useful source of inspiration to come up with examples as well.

In most implementations that rely on an external component to filter and sort records, the functionality would be provided at the interface level (in a SQL query or as arguments to an API, for example). So, it wouldn’t require tests at the unit level at all, and maybe just integration tests. Filtering itself is straightforward. Just use a standard library function like lists:filter/2 and pass in the date with which we want to filter. What’s trickier is ensuring that the predicate passed to the function is correct.

Since the birthday search verifies 366 possible dates, it could be reasonable to just run all of them through an exhaustive search. But in practice, we need to consider leap years, multiple matching employees, and so on. For example, to run an exhaustive search, we would need a list of at least 366 employee birthdays to ensure that at least two employees share a birthday. We would then need to run the program for every day of every year starting in 2022 through 2122, to ensure that each employee is greeted once per year for 100 years on the same day as that employee’s birthday.

This is slightly over one million runs to cover the whole foreseeable future. We can try a sample run to estimate how long that would take, to see if it’s worth the effort. Let’s generate an approximation by running a filter function as many times as we’d need to cover around a hundred years of birthdays for 1,098 employees:

L = lists:duplicate(366*3, #{name => "a", bday => {1,2,3}}),
timer:tc(fun() ->
    [lists:filter(fun(X) -> false end, L) || _ <- lists:seq(1,100*366)], ok 
end).

Test this in the shell of the terminal below. Notice that it takes a while to execute.