Project

General

Profile


Contents page for CI validation documentation:


CI validation debugging

If your CI validation jobs fail, the CI system has a lot of useful ways to get information about the logs and debug, without needing to pull down the .tar of the log files and look through them individually.

The CI dashboard shows you at which stage the process failed:

Hovering over the different boxes can give you some more information. For example, this CI validation submission failed at the "build" stage:

However, note that most (if not all) CI validation builds show failure at the "ci_tests" stage, even if they then go on to successfully complete the "ci_validation" stage. This is nothing to worry about - the CI validation is configured to not run some of the CI "tests" because they are not necessary for our purposes, and that can show up on this system as "failing". In general, it is safe to ignore anything in this column. The three most important columns to look at are "checkout", "build", and "ci_validation".

In the next few sections we will go through the different pieces of information given to you by the CI system. As an example, we will go through how to debug a couple of simple problems (or rather: how to find the correct log files to debug the problems). However, any of these techniques can be used to get information about any problem you have.

If your job fails at "build" stage

Click "show" to expand this section.

If your job fails during the "ci_validation" stage

Click "show" to expand this section.

ci_validationuBooNE.log

Click "show" to expand this section.

FIFEmon and Sandbox

Click "show" to expand this section.

Kibana

Click "show" to expand this section.

Why are my jobs held?

Click "show" to expand this section.

The CI monitoring log

Click "show" to expand this section.

The recovery procedure

Click "show" to expand this section.