Key Principle: Use Automated Tests Created by the Dev Team

ABOUT THIS CHAPTER   Agile development shifted the traditional test emphasis in four ways. First, it increased the emphasis on testing by developers. Second, it emphasized front-loading testing—testing functionality immediately after it is created. Third, it increased emphasis on test automation. Finally, it emphasized testing as a means of refining requirements and design.

These four emphases provide an important safety net for Agile’s other practices, such as just-in-time design and implementation. Without the safety net of a comprehensive automated test suite, the constantly changing design and code environment would give rise to a tidal wave of defects—many of them undetected and going into the latent defect pool described in the “More Effective Agile Quality” chapter. With the automated test safety net, most defects are detected immediately after they are created, supporting the goal of minimizing the gap between defect creation and defect detection.

This chapter describes what we have seen as the most effective test practices for Agile projects.

The development team should be writing automated tests, which are incorporated into an automated build/deploy system. The ideal is using multiple levels and types of testing: API tests, unit tests, integration tests, acceptance tests, UI layer tests, support for mocking, random inputs and data, simulations, etc.

Tests are written by the cross-functional team, which includes developers, testers, or former testers. The ideal is to have developers write unit tests before writing the corresponding code. Test development and automation are inherent aspects of backlog item implementation that are included in effort estimates.

The team should maintain an on-demand test environment that supports automated testing. A combination of automated unit testing (code-level tests) and automated user-level testing should be a core attribute of any Definition of Done.

Developers should be able to test locally, using unit tests and mocking out the behavior of remote systems. A developer should be able to run unit test suites for a complete component of the product within a few minutes, either on a shared team build server or on the developer’s machine.

Local code is promoted to an integration environment, where the developers’ unit tests are aggregated, along with the builds. A team should have the capability to run a complete test pass, including all automated unit- and user-level checks, within 1–2 hours. Many environments measure that time in minutes. The complete test pass should be run multiple times per day.

A sophisticated dev org should have the ability to support continuous integration (CI) by running all automated tests every time a check-in is made. For large projects, that requires numerous virtual environments ganging up to run test suites in parallel, which in turn requires a dedicated team (including test specialists) that builds, maintains, and extends the CI server by incorporating test suites from disparate teams. Large, high-profile companies like Amazon and Netflix are able to support rapid, continuous testing because they have teams that focus solely on this capability, they have invested heavily in computer hardware, and they have developed their capabilities over many years. Companies that are just getting started with CI, or that do not have the needs of an Amazon or Netflix, should scale their expectations appropriately.

Test automation in legacy environments

Inability to develop an ideal test suite should not be taken as a reason not to create automated tests. We’ve seen teams that inherited poor-quality code bases put basic smoke tests in place, slowly backfill automated tests, and realize significant gains from even a small amount of automation. You can support this with a loose DoD that becomes stricter over time.

It works best to focus legacy test work on areas of the code that are being worked on most actively. There’s little benefit in adding coverage for code that’s been stable for years.