There is an update-to-date version of this page at:

NOvA CI test user guide

How to setup the environment

To properly work on the CI system you first need to download and setup the code from the repository

NOTE: Before executing the code, move into the directory where you want to keep the code

source /grid/fermiapp/products/
setup kx509
voms-proxy-init -noregen -rfc -voms fermilab:/fermilab/nova/Role=Analysis

mkdir my_ci_dir
cd my_ci_dir

git clone
svn checkout 

cd generic_ci
setup -. generic_ci

cd ../nova_ci
setup -. -j nova_ci
export NOVA_CI_DIR=${PWD}

The previous code allows you to setup a voms proxy with Analysis role, needed to access some necessary files from the dCache directory.
Then you will download and setup the generic_ci and the nova_ci in the my_ci_dir directory

Tips for a faster setup

The CI environment need to be configured every time you log into the machine.
If you haven't deleted the folder the first time you configured the CI System, you can avoid to download again the code from the repository and use the same code that is on the machine.
If you plan to use the local build more than once, could be good to setup a function into your .bashrc file.
To do that execute the following steps:

NOTE: I'm using vim as text editor but you can open the file with your preferred text editor.

  1. Open the .bashrc file with a text editor
    vim ~/.bashrc
  2. Paste the following code to the end of the file: P.S: In vim you need to press "i" to activate the insert mode
    function setup_ci
        source /grid/fermiapp/products/
        setup kx509
        voms-proxy-init -noregen -rfc -voms fermilab:/fermilab/nova/Role=Analysis
        cd ${my_path}/generic_ci
        setup -. generic_ci
        cd ${my_path}/nova_ci
        setup -. -j nova_ci
        export NOVA_CI_DIR=${my_path}/nova_ci
  3. Save the changes and close the file.
    NOTE: In vim you need to press Esc then type ":wq" and then press Enter to save and exit
Once you completed the previous three steps, every time you log into the machine you can setup the CI Environment using one of the following methods:
  1. giving as argument to the function the full path of the my_ci_dir directory:
    setup_ci <path/of/your/my_ci_dir/directory>
  2. Running the function without argument if you first change directory to the my_ci_dir directory
    cd <path/of/your/my_ci_dir/directory>

Manually trigger a build

Once the environment is set, you can launch the build/workflow executing the trigger script with different options:
  1. To trigger the default CI build
  2. To trigger a CI build with a specific tag for NOvA
    trigger --revisions "novaart.pkgs.svn@S16-03-04"

If you want to trigger your build with mrb or SRT there are two options:

  1. To build with mrb,use the option "--workflow NOvA_CI_MRB"
    trigger --workflow NOvA_CI_MRB
  2. To build with SRT,use the option "--workflow NOvA_CI_SRT"
    trigger --workflow NOvA_CI_SRT

NOTE: As default,trigger use NOvA_CI_SRT as workflow

For the online help page of trigger, use the following command:

trigger -h

Run the CI Tests on the local machine

NOTE: Before to execute the tests locally move to a clean or at least not important folder,considering that running the default test suite at the moment create nearly ten folders in the current directory

To run the CI Tests on the local machine you need to set up the NOvA environment for your local installed NOvA code and the CI environment.

Once the environment is configured, you can see the list of CI Tests and the list of suites available with the following command:

test_runner -l

You can then run the CI tests on the local machine with the following command:

test_runner <name of the test suite or name of the CI test>
for test_runner you can use also the following common options:
  • -v for the verbose mode
  • -d for the debug mode
  • -s to get the statistics of the different tests

so the command will be:

test_runner -v -d -s <name of the test suite or name of the CI test>

For the online help page of the test_runner,use the following command:

test_runner -h

How to add a new CI test.

You can add a new CI test by modifying the following file and commit to SVN (don't forget to commit the file to SVN after modification, otherwise Jenkins won't pick up your new test):


The is file contains all the current CI tests. Each test has a section with configuration parameters, e.g.

[test ci_raw2root_nd_t00_regression_test_novasoft]

args=--nevents %(NEVENTS)s --step %(STEP)s --fhicl %(FHiCL_FILE)s --input %(INPUT_FILE)s --outputs %(OUTPUT_STREAM)s --step-name %(STEP_NAME)s --testmask %(TESTMASK)s

You will need to change "STEP, STEP_NAME, NEVENTS, FHiCL_FILE, INPUT_FILE, REFERENCE_FILE, OUTPUT_STREAM", and if necessary "mem_usage_range" and "cpu_usage_range".

  • STEP: should be a number >=5;
  • STEP_NAME: give a descriptive name of your test;
  • NEVENTS: configure how many events you want to run, this variable is passed to "nova -n";
  • FHiCL_FILE: the name of the FHiCL file for your test, passed to "nova" as argument of "-c";
  • INPUT_FILE: the name of the input file, passed to "nova" as argument of "-s";
  • REFERECEN_FILE: a file you want to compare the the test output file to, e.g. the output file of a successful test job you ran with the same input file and FHiCl file;
  • OUTPUT_STREAM: the output stream of the art job, passed to "nova" as argument of "-o" (don't forget the "out1:" prefix here);

Finally, you will need to create a sub-directory with the name of "STEP_NAME" under:


And put the "FHiCL_FILE, INPUT_FILE, REFERENCE_FILE" under that directory.

The test also needs to be in the "test list" in the following section:

[suite default]
testlist=ci_raw2root_nd_t00_regression_test_novasoft ci_raw2root_nd_t02_regression_test_novasoft ci_raw2root_fd_t00_regression_test_novasoft ci_raw2root_fd_t02_regression_test_novasoft ci_fullchain_nd_data_regression_test_novasoft ci_fullchain_fd_data_regression_test_novasoft ci_calib_nd_regression_test_novasoft ci_calib_fd_regression_test_novasoft

You can run the new test on your local machine now by following the instructions above.
If everything works as expected, you can commit the file "<path/of/your/my_ci_dir/directory>/nova_ci/test/ci_tests.cfg" to SVN.

How to monitor the status of your build.

Please refer to this page