Using CTest to Standardize Testing in CMake

Ultimately, automated testing involves nothing other than running an executable that sets our system under test (SUT) in a given state, performs tested operations, and checks if the results match expectations. We can think of them as a codified way of filling in the blanks in the sentence "GIVEN _ WHEN _ THEN _" and checking if it's true for SUT. As you can imagine, there's more than one way of doing this—actually, there are lots. Everything depends on the kind of framework we're going to use, how we are hooking it up to our SUT, and what is the exact configuration. Even things as minuscule as the filename of our testing binary will impact the experience of the person using our software. As there are no agreed-upon standards for these things, one developer will use the name test_my_app, another will go with unit_tests, and a third will use something obscure or not provide tests at all. Discovering which file needs to be run, which framework is used, which arguments should be passed to the runner, and how to collect results are problems that users would like to avoid.

Executing tests with CTest

CMake solves this by introducing a separate ctest command-line tool. It's configured by the project's author through listfiles and provides a unified way of executing tests: the same standardized interface for every project built with CMake. If we follow this convention, we will enjoy other benefits down the line: adding the project to a (CI/CD) pipeline will be easier, and surfacing them in (IDEs) such as Visual Studio or CLion—all of these things will be streamlined and more convenient. More importantly, we'll get a more powerful test-running utility with very little investment.

How to execute tests with CTest on an already configured project? We'll need to pick one of the following three modes of operation:

• Test

• Build-and-test

• Dashboard client

The last mode allows us to send the results of the test to a separate tool called CDash (also from Kitware). CDash collects and aggregates software-quality test results in an easy-to-navigate dashboard, as illustrated in the following screenshot:

CDash is outside the scope of this course since it's an advanced solution used as a shared server, accessible for all developers in a company.

Note: If you're interested in learning more online, reference the official documentation of CMake and visit the CDash website.

Let's get back to the first two modes. The command line for test mode looks like this:

In this mode, CTest should be executed in the buildtree, after building the project with cmake. This is slightly cumbersome during the development cycle, as we'd need to execute multiple commands and change the working directory back and forth. To simplify the process, CTest added a second mode: build-and-test mode.

Build-and-test mode

To use this mode, we need to execute ctest starting with --build-and-test, as follows:

Essentially, this is a simple wrapper around the regular test mode that accepts a few build configuration options and allows us to append the command for the first mode—in other words, all options that can be passed to ctest <options> will work when passed to ctest --build-and-test. The only requirement here is to pass the full command after the --test-command argument. Contrary to what we might think, build-and-test mode won't run any tests unless provided with ctest keyword after --test-command, like so:

In this command, we specify source and build paths and select a build generator. ...