Project

General

Profile

History

Wiki » Uboone guide »

Unit Tests and Integration Tests

Unit tests and integration tests are tests that run during the mrb build process. Conceptually, a unit test runs on the smallest possible unit of code (typically, a single c++ class or source file, not necessarily an art object like an art module). Conceptually, an integration test tests multiple units of code. in practice, a unit test runs out of the build area, whereas an integration test runs out of the install area.

Invoking Tests

Either type of test can be invoked by typing "mrb test" or "make test" in the mrb build area.

Directory structure

The standard directory structure in the mrb srcs area is to have a test subdirectory in the top level of each package (so that tests are in a completely separate directory tree than production code). The directory structure below the test directory is arbitrary. Add the following directives in the top level CMakeLists.txt file.

include(CetTest)
add_subdirectory(test)

Also put add_subdirectory directives in lower level CMakeLists.txt files to reach the entire test directory tree.

Configuring Tests

Either type of test is configured by adding a directive in CMakeLists.txt that looks like this.

cet_test( mytest ...)

Here mytest is the name of the test, which can be the name of a stand alone executable, or the name of a script, or simply an invented name. Full documentation of the cet_test cmake directive can be found in $CETBUILDTOOLS_DIR/Modules/CetTest.cmake. Note that cetbuildtools ups product is only set up in an initailized build area (after typing "mrbsetenv"). Additional details about configuring unit tests and integration tests can be found in the following sections.

Unit Tests

A unit test is a stand alone test executable that links against the package's shared libraries. The test executable decides whether the test is successful and returns the result as an exit status (0=success). The configuration of a simple unit test can look like this in CMakeLists.txt.

cet_test( mytest LIBRARIES mylib )

Here mytest is the name of a c++ source file (i.e. mytest.cc) containing a c++ main program. The LIBRARIES keyword defines which libraries the test executable needs to link against, which would typically include (one of) the package's shared libraries (i.e. libmylib.so). The test executable, called mytest, is built and placed in the build area under $MRB_BUILDDIR/<package_name>/bin, from which location it can be run outside the build system, if necessary (for example, in a debugger).

Assertions in unit tests.

It is perfectly acceptable to use assertions inside unit tests as a way of indicating failure. In this case, you should make sure that assertions are enabled in CMakeLists.txt using this directive.

cet_enable_asserts()

In most cases, this directive should be in all top level test CMakeLists.txt files, and is automatically inherited to subdirectories.

Boost unit tests.

The boost product provides a framework for unit tests consisting of c++ macros and libraries. To enable boost unit tests, use keyword USE_BOOST_UNIT in your cet_test directive.

cet_test( mytest USE_BOOST_UNIT LIBRARIES mylib )

Integration Tests

The basic idea of an integration test is to get cmake to run an installed executable (i.e. lar) with specified command line options, using installed shared libraries, and to report a result. There are two ways to do this: with or without a script.

Integration tests without a script

With this type of integration test, you can run a lar job, and the only indication of whether the test succeeded is the exit status of lar. The cmake configuration in CMakeLists.txt for this type of integration test looks something like this.

cet_test( mytest HANDBUILT 
          TEST_EXEC lar
          TEST_ARGS -c mytest.fcl )

In this case, cmake will create a test area in $MRB_BUILDDIR/<package name>/test/<src path>/mytest.d). The test command will be run from there (i.e. lar -c mytest.fcl), and any files generated by the test job will be in the test area after running the test. The package name is the name of the top level package (e.g. uboonecode), and the source path will match the path in the mrb srcs area.

Integration tests with a script

Sometimes you want a more sophisticated determination of success than simply whether lar crashed or not. For example, you might want to examine an output file or do tests involving generated histograms, etc.. In these cases, you should write a test script to run lar and determine whether the test was successful.

Your test script (mytest.sh) should look like this.

#! /bin/bash

lar -c mytest.fcl > mytest.out 2> mytest.err
stat=$?
echo "lar exited with status ${stat}." 
if [ $stat -ne 0 ]; then
  exit $stat
fi

# Do additional tests.
 .
 .
 .
exit 0   # Success.

And the CMakeLists.txt configuration should look like this.
cet_test( mytest.sh PREBUILT )

In this case, the test directory in the build area will be called $MRB_BUILDDIR/<package name>/test/<src path>/mytest.sh.d). In this example, mytest.out and mytest.err will be found there.

Test dependencies.

In general unit and integration tests are allowed to depend on any external or larsoft product, provided that the shared libraries need to link and/or run are available in $LD_LIBRARY_PATH.

Integration tests will often depend on microboone-specific resources from uboonecode (e.g. microboone geometry or microboone fcl files). An integration test that depends on uboonecode can not be included in any general larsoft product. Any integration test that depends on uboonecode in any way will fail during the larsoft build, because the larsoft build does not set up uboonecode. For this reason, most integration tests should go in the uboonecode package.

Tests with data files.

For tests that require additional data files use keyword DATAFILES in the cet_test directive.

cet_test( mytest ... DATAFILES file1.dat file2.dat ... )

The specified data files will be copied to the test area before running the test.