Stateful Property-Based Testing

Until now, we’ve mostly looked at property-based testing in the context of testing pure, stateless functions that take an input and return an output. However, property-based testing is also useful for testing stateful systems.

What is the stateful system?

A stateful system is a system that, well, carries the state. For example, a database is a stateful system.

In our examples so far, we only used property-based testing to generate some data and then feed it to a piece of code and assert the result of that. With stateful systems, things change: we now have to deal with setting a state and only executing some operations when the system is in a given state. Let’s see how we can use property-based testing for something like that.

Modeling the Stateful System

We know how to generate random data through our property-based testing framework. We can take advantage of this knowledge to generate random commands that we can issue on our stateful system.

For example, if our stateful system is a database, we can generate random commands to issue against this system. However, if the commands are random, how do we assert their effects on the system? Enter the system model. The whole idea behind property-based testing of a stateful system revolves around the idea of modeling the real system with a model that represents that system from the perspective we’re interested in. Once we have this model, we can execute the commands we generated on the real system and on the model and then verify that the effects of the commands match. This sounds complex, so let’s break it down with an example.

Let’s imagine we wrote a key-value store where we can write values stored under unique keys and retrieve those values using the corresponding keys.

iex> kv_store = KVStore.new()
iex> KVStore.set(kv_store, "key1", "some value") 
iex> KVStore.get(kv_store, "key1")
"some value"
iex> KVStore.delete(kv_store, "key1")
iex> KVStore.get(kv_store, "key1")
nil

This is a stateful system where the state is the set of key-value pairs stored in the key-value store created with KVStore.new/0. One thing we could test about this system is that retrieving an existing key always returns the last value set for that key or a null value if the last operation was to delete the value, no matter how many times we set or delete that key. How can we model our key-value store in order to test this property? We can start with a simple two-element tuple {key, value} that stores the key we’re interested in and its current value (or nil if the key hasn’t been set yet).

Let’s use pseudocode to see how we could generate a command assuming we’ve already generated a random key called random_key: