RSpec or Minitest Bisect

RSpec and Minitest bisecting

Minitest and RSpec both have their own version of bisect, which solves a slightly different problem. Sometimes we’ll have a test failure caused by the order in which the tests are run. This usually means the tests aren’t completely isolated, and one test is changing the global state in a way that breaks another test. This cannot be easy to reproduce, especially if the test suite is long.

The bisect option for Minitest and RSpec gives us a minimal set of specs that reproduces the error so that we can run the minimal set from the command line and reproduce the problem efficiently. Once the minimal set is identified, it’s much easier to find the problem.

Rspec and Minitest commands

The two commands work almost identically, which is not surprising since RSpec explicitly cribbed the feature from Minitest. The Minitest version does require a gem, minitest-bisect, to be installed.

Test suite

To make this work, we need a test suite that has an intermittent failure that we believe is due to test ordering, and we need a seed value that produces a failure. In RSpec, if we’re not already running the tests in a random order, add the line --order rand to the .rspec file. Each run will then tell us what seed value RSpec used for the randomization Randomized with seed 57180. We can then reproduce the exact ordering with rspec --seed 57180. Minitest behaves similarly, except that the random ordering is the default behavior.

Once we have a seed that consistently produces an ordering with a test failure, we trigger the bisect. In RSpec, that’s done with the following:

$ bundle exec rspec --seed 57180 --bisect

Let’s run the command in the terminal: