CMake CTest subset test label

The CMake test frontend CTest can easily select subsets of tests. While there are more advanced CTest test selection options, two of the most common and easy to use test subset selection methods are by regex selection of names, labels and/or fixtures exclusion. Assuming the project has a meaningful test naming scheme, one may trivially select tests by either or both of the ctest -R and ctest -E flags. For this article, assume the tests are named:

alpha:egg
alpha:bacon
beta:egg
beta:apple
gamma:egg
gamma:orange

where each test was setup in CMakeLists.txt like:

add_executable(alpha_egg alpha_egg.c)

add_test(NAME alpha:egg COMMAND alpha_egg)

The colon has no special meaning, and CTest names may use special characters if desired. When figuring out how to use CTest test selection, it’s very helpful to also add the ctest -N option, so that test names are printed without running the tests. For all examples, we assume the user working directory is PROJECT_BINARY_DIR or is using ctest --test-dir.

One may select all the “egg” tests by:

ctest -R egg

Suppose one wishes to exclude the test named “beta:egg”:

ctest -R egg -E beta

To run all tests except those with “beta” in the name:

ctest -E beta

A more sophisticated test selection scheme requires setting test labels in the respective CMakeLists.txt like:

set_property(TEST alpha:egg beta:apple gamma:egg PROPERTY LABELS "unit;gravy")

Combinations of test labels and test names regex can be used to select subsets of tests. For example:

ctest -R egg -L unit

Note also the option ctest -LE, which works like ctest -E for labels.

CTest fixtures can also be excluded with the ctest -FA option. This allow not rerunning expensive FIXTURES_SETUP tests when not needed.

Set labels for all targets and tests in a directory like:

set_property(DIRECTORY PROPERTY LABELS linalg)

where “linalg” is the desired label(s).