Why Speed Is Important

Speed as a measure of quality

Before we explain how we’ll use this information to speed up our tests, let me make a pitch for the idea of being a little obsessive about test speed.

We’re going to run our tests a lot.

If we’re doing the kind of rapid-feedback TDD that we’ve been discussing up until now, we’ll be running at least a subset of our tests nearly continuously, multiple times an hour. (Any exact number that we put here is going to seem shockingly high to people who aren’t used to super-fast test feedback and shockingly low to those who are). And we’re probably not the only developer on our team. Also, our entire test suite likely runs when we commit code to our repository via some continuous build system.

Speed test groups

The speed of a test run, whether it is our entire suite or a subset we’re running in development, lands in one of seven groups:

1. So fast that we don’t even notice that the test ran
2. Fast enough for us to watch the test run without breaking focus
3. Slow enough that we break focus but don’t have time to do anything else
4. Slow enough for us to get a cup of coffee
5. Slow enough for us to eat lunch
6. Slow enough for us to sleep
7. Effectively infinite without a big parallelized server

A project will have thousands of runs. Tens of thousands. Hundreds of thousands if the project is big. The difference between being in group 2 and group 3 becomes a big deal when we’re trying to run the tests as often as possible. Numerous studies say that breaking focus is a huge drain in developer productivity.

Example

The difference between a 10-second run in group 2 and a 30-second run in group 3 is important. A group 2 run allows us to get feedback without breaking focus. A group 3 run means we lose our train of thought. If we’ve ever been able to work in an environment with near-instantaneous test feedback, we know it’s an entirely different level of dealing with tests.

Worse, there’s a slow acclimation that comes with a gradually increasing test suite. We’ll add only a little bit of time to the test suite every day, practically nothing. But weeks and months of practically nothing being added lead us not to notice that suddenly we have time to get coffee during our test run. Preventing this gradual time creep takes vigilance.

We’re not saying that our entire test suite needs to run in 15 seconds. That’s not feasible for most applications, and it’s probably not a good idea in any case. We need some of those slow integration tests. What we’re saying is that our testing value will improve if we have a reliable way to run relevant tests in a few seconds. That way might be Guard, it might include Spring, or RSpec metadata, or the --only-failures options, but it’s worth making sure we have a way of doing so. It’s also true that it’s much easier to isolate a subset of tests if our application is broken into relatively distinct sections.

Note: If we have enough time to break focus while our tests run, we aren’t getting the full value of the TDD process.