Well Defined: The Second Quality of Valuable Tests

Well defined

A test is well-defined if running the same test repeatedly gives the same result. If the test is not well defined, the symptom will be intermittent, seemingly random test failures (sometimes called Heisenbugs, Heisenspecs, or Rando Calrissian).

Repeatability problems

The three classic causes of repeatability problems are:

• Time and date testing
• Random numbers
• Third-party or Ajax calls

These all have the same root cause: ensuring that each test is run in a consistent environment is impossible. Dates and times have a nasty habit of never staying the same. The time is continually increasing, while random data stubbornly insists on being random. Similarly, tests that depend on a third-party service or even test code that makes Ajax calls back to our application can vary from test run to test run, causing intermittent failures.

Dates and times tend to lead to intermittent failures when certain magic time boundaries are crossed. We can also get tests that fail at particular times of the day or run in certain time zones. In contrast, random numbers make it somewhat hard to test the randomness of the number and that the arbitrary number is used properly in whatever calculation requires it.

Test plan

The test plan is similar for dates, randomness, and external services. It applies to any constantly changing dataset. The key is to make the test data replicable and consistent. We can do this with a combination of encapsulation and test doubles. We encapsulate the data by creating a service object that wraps around the changing functionality. We make it easier to stub or mock the output values by mediating access to the changing functionality. Stubbing the values provides the consistency we need for testing. We provide the exact value used in the test.

Example test

We might create a RandomStream class that wraps Ruby’s rand() method:

class RandomStream
  def next
    rand()
  end
end

This example is a little oversimplified. Normally, we would encapsulate RandomStream. We can provide more specific methods tuned to the use case with the wrapper class, something like def random_phone_number. First, we unit-test the stream class to verify that the class works as expected. Any class that uses RandomStream can be provided with mock random values to allow easier and more stable testing.

ActiveSupport gem methods

The exact mix of encapsulation and mocking varies. The Rails ActiveSupport gem has methods that stub the time and date classes with no encapsulation. That allows us to specify an exact value for the current time for testing purposes.

In Testing External Services, we’ll learn this pattern for wrapping a potentially variable external service in more detail. We’ll cover mock objects in Using Test Doubles as Mocks and Stubs, and we’ll learn more about debugging intermittent test failures in Troubleshooting and Debugging.