How Do You Know the Tests Are Right?

This is an objection that has merit, so we need to deeply understand the logic behind it. This is a common objection from people unfamiliar with writing automated tests because they misunderstand how we avoid incorrect tests.

By helping them see the safeguards we put in place, we can help them reframe their thinking.

Understanding the concerns behind writing broken tests

One objection we’ll hear is,

“How do we know the tests are right if the tests themselves don’t have tests?”

This objection was raised the first time we introduced unit tests to a team. It was polarizing. Some of the team understood the value right away. Others were indifferent, but some were actively hostile. They saw this new practice as suggesting they were somehow deficient. It was perceived as a threat. Against that background, one developer pointed out a flaw in the logic we had explained.

Author’s experience: visual inspection

We told the team that we could not trust our visual reading of production code. Yes, we’re all skilled at reading code, but we’re humans, so we miss things. Unit tests would help us avoid missing things. One bright developer asked a great question: if visual inspection does not work for production code, why are we saying that it does work for test code? What’s the difference between the two?

The right illustration for this came after we needed to test some XML output. The code we had written for checking the XML output was truly complex. The criticism was correct. There was no way we could visually inspect that test code and honestly say it was without defects.

So, we applied TDD to the problem. We used TDD to write a utility class that could compare two XML strings and report either that they were the same or what the first difference was. It could be configured to ignore the order of XML elements. We extracted this complex code out of my original test and replaced it with a call to this new utility class. We discovered the utility class did not have any defects because it passed every TDD test that we had written for it. There were many tests, covering every happy path and every edge case we cared about. The original test that had been criticized now became very short and direct.

We asked my colleague who had raised the point to review the code. They agreed that in this new, simpler form, they were happy to agree that the test was correct, visually. They added the caveat “if the utility class works right.” Of course, we had the confidence that it passed every TDD test we had written it against. We were certain that it did all the things we specifically wanted it to do, as proven by tests for these things.