Test Harnesses

Integration testing

As we’ve seen in previous chapters, distributed systems have failure modes that are difficult to provoke in development or QA environments. To be more thorough about testing various components together, we often resort to an “integration testing” environment. In this environment, our system is fully integrated to all the other systems it interacts with.

Integration testing presents problems of its own, however. What version should we test against? For greatest assurance, we’d like to test against the versions of our dependencies that will be current when we release our system. We could prove by induction that this approach constrains the entire company to testing only one new piece of software at a time. Naturally, the proof itself is left as an exercise for the reader. Furthermore, the interdependencies of today’s systems create such an interlocking web of systems that an integration testing environment really becomes unitary. One global integration test that duplicates the real production systems of the entire enterprise. Such a unitary environment would need change control just as rigorous, or perhaps more so, than the actual production environments. There is a more abstract difficulty. Integration test environments can verify only what the system does when its dependencies are working correctly.

Although it may be possible to provoke the remote system into returning errors, it’s still functioning more or less within specifications. If the specifications say, ”The system shall return an error code 14916 unless the request includes the date of the last telephone sanitization,” then the caller can force that error condition to occur. Nevertheless, the remote system is still operating within specifications.

Test the local system behavior

The main theme of this course, however, is that every system will eventually end up operating outside of spec. Therefore, it’s vital to test the local system’s behavior when the remote system goes wonky. Unless the designers of the remote system built in modes that simulate the whole range of out-of-spec failures that can occur naturally in production, there will be behaviors that integration testing does not verify.

Enhance system isolation

A better approach to integration testing would allow you to test most or all of these failure modes. It should preserve or enhance system isolation to avoid the version-locking problem and allow testing in many locations instead of the unitary enterprise-wide integration testing environment I described earlier. To do that, you can create test harnesses to emulate the remote system on the other end of each integration point.

Test harnesses

Hardware and mechanical engineers have used test harnesses for a long time. Software engineers have used test harnesses, but not as maliciously as they should. A good test harness should be devious. It should be as nasty and vicious as real-world systems will be. The test harness should leave scars on the system under test. Its job is to make the system under test cynical.