Software executable dry run

Developers covering multiple platforms and archs can benefit from including a self-contained dry run. We define a software dry run as a fast self-contained run of the executable, exercising most or all of the program using actual input files. The concept of dry run is used by popular programs that rely on several components and connections including rsync.

A dry run self-check can be used from Python or any other script calling the executable to ensure the binary is compatible with the current platform environment. The dry run helps mitigate confusing error messages by checking that the executable runs on the platform before making a large program run.

The dry run can catch platform-specific issues like:

  • incompatible executable format (running a executable built for another platform)
  • executable built for incompatible arch (using CPU feature not available on this platform)
  • shared / dynamic library run-time search path issues

The dry run does not output any files besides temporary files. For example, in a simulation, the dry run might run one complete time step. To test file I/O, optionally write temporary file(s) using the same file format. An advanced dry run might read in those temporary files and do a basic sanity check.

A dry run is distinct from an integration test. A dry run of the program just checks that the platform environment is OK to run with this binary. The dry run checks simply that the code executes without crashing. The dry run does not emphasize deep checks of program output as an integration test would. Consider making the dry run return code be 0 for compatibility with CMake and other high level build systems.

Implement a check of the dry run with CMake:

project(Foo LANGUAGES C)
enable_testing()

add_executable(foo foo.c)

add_test(NAME check_foo COMMAND foo -dryrun <other command line flags>)
set_property(TEST check_foo PROPERTY PASS_REGULAR_EXPRESSION "OK: myprogram")

PASS_REGULAR_EXPRESSION to verify the special dry run text you put in the executable code. The dry run test normally returns code zero, but PASS_REGULAR_EXPRESSION ignore the executable return code.