Project

General

Profile

g-2 Continuous Integration (gm2 CI)

How to setup and trigger a CI build

How to setup the environment

To properly work on the CI system you first need to checkout and/or setup the CI code from the repository

NOTE: Before executing the code, move into the directory where you want to keep the code

source /cvmfs/fermilab.opensciencegrid.org/products/common/etc/setups.sh
source /cvmfs/gm2.opensciencegrid.org/prod/g-2/setup
setup git

mkdir my_ci_dir
cd my_ci_dir

kinit
git clone ssh://p-gm2_ci@cdcvs.fnal.gov/cvs/projects/gm2_ci
cd gm2_ci
setup -. gm2_ci
git checkout master

The previous code allows you to setup the gm2_ci local UPS product.

Be sure to source the two setup files in the order shown above. This will ensure that UPS gives you a sufficiently recent version of git (probably 2.7 or higher). You may choose to set up the UPS products for gm2 v7 (using /cvmfs/gm2.opensciencegrid.org/prod7/g-2/setup) or the v8/latest products (as shown above).

Manually trigger a build

Once the environment is set, you can trigger the CI build executing the trigger script with different options.

The inline help page of trigger is available using the following command:

trigger -h
  1. To trigger the default CI build
    trigger
  2. To trigger a CI build with a specific tag
    trigger --revisions "repo1@TAG1 repo2@TAG2"

where "repoN" is the name of the repository, and "TAG" can be a tag, a revision, a branch for the associated repo.

Note that there is a delay before the job starts where it will not show up in the monitoring application. This delay is currently set to 15 minutes.

If you want to trigger your build with a specific workflow:

trigger --workflow gm2_defaultwf 

This triggers a CI build using the workflow gm2_defaultwf that tests all g-2 software modules.

An email report is sent also to whom manually triggered the CI build, besides to the other mailing list configured in the workflow.cfg.

Run the CI Tests on the local machine

NOTE: Before to execute the tests locally move to a clean directory, considering that running CI tests you get a new folder for each CI test you are running. Also make sure to have access to enough disk space, depending on the nature of the CI tests executed, you could need few Gb of disk space.

To run the CI Tests on the local machine you need to set up the environment for your local installed g-2 code and the CI environment.

Once the environment is configured, you can see the list of CI Tests and the list of suites available with the following command:

test_runner -l

You can then run the CI tests on the local machine with the following command:

test_runner <name of the test suite or name of the CI test>
for test_runner you can use also the following common options:
  • -v for the verbose mode
  • -d for the debug mode
  • -s to get the statistics of the different tests

so the command will be:

test_runner -v -s <name of the test suite or name of the CI test>

As default test_runner process up to 5 tests in parallel, this value can be modified using the option --parallel-limit.

For the online help page of the test_runner, use the following command:

test_runner -h

How to monitor the status of g-2 CI builds

The gm2_CI Web Application

To monitor the status of your builds into the CI system, our team created an experiment based web application to help you with your monitoring job.

The application can be found at the following link:

http://gm2-ci-history.fnal.gov/Gm2CI/app

This application offer to you all the necessary tools to monitor the status and the performance of your builds in a short and long term.
It offers a quick overview of the status of the last ten builds and offers also dynamic graphs to show the overtime performance of the different build phases.

The main page of the web application

g-2 CI web app

The previous image shows the g-2 CI web application.

This page is divided into three main sections:
  1. The search and filter section ( on the left of the image )
  2. The legend section ( on the right of the image )
  3. The CI build status section (in the middle of the image )

The search and filter section

The purpose of this section is to offer to the user a tool to filter the results shown in the web application.
This panel allows you to:
  • Show the builds up to a defined build number.
  • Show the builds in a defined time range.
  • Filter the builds for platform or for test suite used.

The legend section

This section show the association between the color of the different phases of the builds,and their status
  • [#] yellow -> The phase is running
  • [#] gray -> The phase is pending
  • [#] green -> The phase was successful
  • [#] orange -> The phase failed in warning status
  • [#] red -> The phase failed
  • [#] light blue -> The phase have been skipped
    NOTE: A warning status is when the tests are successful but the product size or names are different when the output is compared to a reference file

The CI build status section

This is the main section of the CI web app and it's where the most important data is shown.
This section shows a table of the last ten CI builds executed and their corresponding information.

The table is divided into multiple columns.

The first three columns show some basic information about the CI build number, the time when the CI build started and the platform on which the CI build have been executed.

The next columns show the status of the different phases of the CI build.
Each phase is represented by a square with a bullet

In this moment a g-2 CI build is divided into five phases:
  1. Checkout
    Show the status of the checkout of the code from the repositories.
  2. Build
    shows the status of the building process of the code into the CI System.
  3. Unit Test
    Show the progress of the configured unit tests.
  4. Install
    Show the status of the install process of the code into the CI System.
  5. CI tests
    Show the progress of the CI tests defined for this build.

Dynamic graphs and statistics

Every bullet and the build number into the web application is clickable.
Performing a click on a bullet open a different web page that shows detailed information on the phase.

Build Details

Checkout Phase Details

Build Phase Details

Unit Tests Details

Install Phase Details

CI Tests Phase Details

Links

https://cdcvs.fnal.gov/redmine/projects/gm2_ci

https://cdcvs.fnal.gov/redmine/projects/ci

https://buildmaster.fnal.gov/me/my-views/view/All/job/gm2_ci/ (requires authentication on Jenkins)

http://gm2-ci-history.fnal.gov/Gm2CI/app