History

LArSoft Continuous Integration (LArCI)¶

Below is the old wiki page - to be removed

How to setup and trigger a CI build¶

How to setup the environment¶

To properly work on the CI system you first need to checkout and/or setup the code from the repository

NOTE: Before executing the code, move into the directory where you want to keep the code

source /cvmfs/fermilab.opensciencegrid.org/products/common/etc/setups.sh
kinit
setup cigetcert
cigetcert -s fifebatch.fnal.gov
voms-proxy-init -noregen -rfc -voms fermilab:/fermilab/<experiment>/Role=Analysis

# for DUNE use
# voms-proxy-init -noregen -rfc -voms dune:/dune/Role=Analysis

mkdir my_ci_dir
cd my_ci_dir

setup lar_ci

NOTE: Make sure to replace <experiment> with the experiment VO you are affiliated

The previous code allows you to setup a voms proxy with Analysis role, needed to access files required to run CI tests from the dCache directory.
Then you setup generic_ci UPS product then setup the lar_ci.

Tips for a faster setup¶

The CI environment needs to be configured every time you log into the machine.
If you haven't deleted the folder used the first time to configure the CI System, you can avoid to checkout again the code from the repository and use the existing code on the machine.
If you plan to use the local build more than once, could be good to setup a function into your .bashrc file.
To do that execute the following steps:

NOTE: I'm using vim as text editor but you can open the file with your favorite text editor.

Open the .bashrc file with a text editor
```
vim ~/.bashrc
```

Paste the following code to the end of the file: P.S: In vim you need to press "i" to activate the insert mode

function setup_ci
{
    my_path="${1:-${PWD}}" 

    source /cvmfs/fermilab.opensciencegrid.org/products/common/etc/setups.sh
    kinit
    setup cigetcert
    cigetcert -s fifebatch.fnal.gov
    voms-proxy-init -noregen -rfc -voms fermilab:/fermilab/uboone/Role=Analysis

    setup lar_ci
}

Save the changes and close the file.
NOTE: In vim you need to press Esc then type ":wq" and then press Enter to save and exit

Once you completed the previous three steps, every time you log into the machine you can setup the CI Environment using one of the following methods:

giving as argument to the function the full path of the my_ci_work_dir directory:
```
setup_ci <path/of/your/my_ci_work_dir/directory>
```
Running the function without argument if you first change directory to the my_ci_work_dir directory
```
cd <path/of/your/my_ci_work_dir/directory>
setup_ci
```

Manually trigger a build¶

Once the environment is set, you can trigger the CI build executing the trigger script with different options.

The inline help page of trigger is available using the following command:

trigger -h

To trigger the default CI build
```
trigger
```
To trigger a CI build with a specific tag/branch/revision
```
trigger --revisions "repo1@TAG1 repo2@TAG2"
```

where "repo" is the name of the repository, and "TAG" can be a tag, a revision, a branch for the associated repo.
The default branch used is "develop".
You can also use the "*" wildcard to select repositories, i.e. "*@feature/testbranch" will use the "feature/testbranch" for all used repositories.
Note that, when using wildcard, the first matching "repository@revision" is used.

If you want to trigger a CI build with a specific workflow:

trigger --workflow defaultwf

This triggers a CI build using the workflow defaultwf that tests LArSoft + ArgoNeuT + DUNE + LArIAT + uBooNE + SBND.
The workflow configuration has also dedicated workflows to test individual exp code standalone or together with LArSoft.
The list of currently available workflows is available here.

An email report is sent also to whom manually triggered the CI build, besides to the other mailing list configured in the workflow.cfg.

Manual trigger to generate a new set of reference files¶

When you have the CI environment set up you can trigger a manual CI build to generate a new set of reference files for a specific list of CI tests.

trigger --build-delay 0 --ci-tests "list of CI tests that require new reference files" --workflow <Workflow Name>

where the "list of CI tests that require new reference files" uses white space as separator and should be included within double quotes. If the option is missed a "default" CI test suite will be used.

The pattern of the reference files is: <filename_prefix>_Reference-<platform>.<filename_suffix>.root
The current <platform> is slf6 and it depends on the node where the CI build runs.

~~When generating a new set of reference files, the system is forced to process only 1 event.~~
If the experiment code fails to generate the output file, the CI phase is declared Failed and the reference files are not uploaded in the experiment dCache area.

NOTE: If the experiment code fails to generate the output file, CI phase is declared Failed and the reference files are not uploaded in the experiment dCache area.
So if there are CI tests known to fail, take the failing test out of the CI tests list. You can select also a single CI test for which generate new reference files.
This feature allows you to update reference files also if some of the CI test is failing, and only for CI tests that you really need to update references files.

The procedure to generate the new set of reference files copies the new reference files in the related STAGE_NAME folder in dCache in the form <filename_prefix>_Reference-<platform>-<timestamp>.<filename_suffix>.root
These files are tested using the CI tests, if the CI tests pass, the new reference files are copied as <filename_prefix>_Reference-<platform>.<filename_suffix>.root keeping a copy of the new reference file with the time stamp.
The previous reference files are copied in backup files, just in case the new reference file has some issue.
The existing backup file is removed.

The current list of available reference files is:

Update_ref_files_DUNETPC
Update_ref_files_DUNETPC_standalone
Update_ref_files_UBOONE_standalone
Update_ref_files_ARGONEUT_standalone
Update_ref_files_LARIAT_standalone

Manual trigger for a CI validation workflow¶

CI validation uses specific workflows that build the required version of the code, usually a new tag or the develop branch, and uses it to process an experiment workflow.
The usual trigger command is in the form:

#For uBooNE
trigger --build-delay 0 --version develop --workflow CI_VALIDATION_UBOONE --gridwf-cfg cfg/grid_workflow_uboone_datamc_cosmic.cfg,cfg/grid_workflow_uboone_mc_bnb.cfg --force-platform slf6

This command triggers two CI builds, one for each of the validation workflows provided through the --gridwf-cfg option.
The code will be built only on the SLF6 platform (--force-platform slf6).

As for the standard CI builds the user can choose a code tag/branch/revision (option --revisions)

trigger --help

provides the full list of available options.

If configured, when the CI validation is completed, a Slack notification is sent with a link to the CI validation dashboard where the validation results are available.

NOTE: The user needs to provide a valid proxy to be able to run the CI build and submit the CI validation jobs.
The proxy can be generated using the commands:

cigetcert -s fifebatch.fnal.gov
voms-proxy-init -rfc -noregen -voms fermilab:/fermilab/uboone/Role=Analysis -valid 120:00

Run the CI Tests on the local machine¶

NOTE: Before to execute the tests locally move to a clean or at least not important folder, considering that running CI tests you get a new folder for each CI test you are running. Also make sure to have access to enough disk space, depending on the nature of the CI tests executed, you could need few Gb of disk space.

To run the CI Tests on the local machine you need to set up the environment for your local installed LArSoft and/or experiment code and the CI environment.

Once the environment is configured, you can see the list of CI Tests and the list of suites available with the following command:

test_runner -l

You can then run the CI tests on the local machine with the following command:

test_runner <name of the test suite or name of the CI test>

for test_runner you can use also the following common options:

-v for the verbose mode
-d for the debug mode
-s to get the statistics of the different tests

so the command will be:

test_runner -v -d -s <name of the test suite or name of the CI test>

As default test_runner process up to 5 tests in parallel, this value can be modified using the option --parallel-limit.

For the online help page of the test_runner, use the following command:

test_runner -h

How to monitor the status of your build¶

The Lar_CI Web Application¶

To monitor the status of your builds into the CI system, our team created an experiment based web application to help you with your monitoring job.

The application can be found at the following link:

http://lar-ci-history.fnal.gov/LarCI/app

This application offer to you all the necessary tools to monitor the status and the performance of your builds in a short and long term.
It offers a quick overview of the status of the last ten builds and offers also dynamic graphs to show the overtime performance of the different build phases.

The main page of the web application¶

CI System Homepage

The previous image shows the homepage of the LarCI web application.

This page is divided into three main sections:

The search and filter section ( on the left of the image )
The legend section ( on the right of the image )
The Show data section (in the middle of the image )

The search and filter section¶

The purpose of this section is to offer to the user a tool to filter the results shown in the web application.
This panel allows you to:

Show the builds up to a defined build number.
Show the builds in a defined time range.
Filter the builds for platform or for test suite used.

The legend section¶

This section show the association between the color of the different phases of the builds,and their status

[#] yellow -> The phase is running
[#] gray -> The phase is pending
[#] green -> The phase was successful
[#] orange -> The phase failed in warning status
[#] red -> The phase failed
[#] light blue -> The phase have been skipped
NOTE: A warning status is when the tests are successful but the product size or names are different when the output is compared to a reference file

The Show data section¶

This is the main section of the homepage and it's where the most important data is shown.
This section shows a table of the last ten builds executed and their corresponding information.

The table is divided into multiple columns.

The first three columns show some basic information about the CI build number, the time when the CI build started and the platform on which the CI build have been executed.

The next columns show the status of the different phases of the CI build.
Each phase is represented by a square with a bullet

In this moment a Lar CI build s divided into five phases:

Checkout
Show the status of the checkout of the code from the repository.
Build
shows the status of the building process of the code into the CI System.
Unit Test
Show the progress of the unit tests configured.
Install
Show the progress of the install process of the code into the CI System.
CI tests
Show the progress of the CI tests defined for this build.

Dynamic graphs and statistics¶

Every bullet and the build number into the web application is clickable.
Performing a click on a bullet open a different web page that shows detailed information on the phase.

Project

General

Profile

Continuous Integration (CI) » lar_ci

Wiki

LArSoft Continuous Integration (LArCI)¶

How to setup the CI environment¶

How to trigger the standard CI build¶

How to run CI tests interactively¶

How to trigger the CI Validation build¶

How to monitor the status of CI builds¶

LArCI Workflows¶