Project

General

Profile


Contents page for CI validation documentation:


How to tell whether a CI validation build has worked or failed

When CI validation builds are triggered automatically based on the git commit messages when the code is tagged, we will need to monitor the jobs to ensure that the build is successful. There are two points at which a build can fail, and the way to tell that the build has failed will depend on when it happened:

  1. The CI build fails before it gets to the stage of launching jobs. Every time a new CI build is triggered, it should appear on the CI dashboard (this link includes both Cosmic and BNB validations) within 15 minutes of being triggered. If, after more than 15 minutes, you do not see a new entry on the CI dashboard (check the time stamps in the "Start Time" column) then it is likely that the build has failed.
  2. The CI build fails while running. If the build gets to the point that it appears on the CI dashboard, then you can monitor its progress there.
    1. Look for whether you see a green check or a red cross in the "build" column (see the example screenshot below).
    2. Another way to check the status of the build: If you follow the instructions to view the validation plots, you will notice that the status of each stage appears at the top of the page. An example of this is shown below. You can see that in total 9,950 events passed the simulation stage, and 9,400 events passed the reconstruction stage. As shown in the second example screenshot, hovering your mouse over one of the stages will show you the fates of the jobs at that stage. Here you can see that 9 of the reco jobs failed, 1 was skipped, and 2 were held. In total 188 jobs completed successfully.
    3. One final way to tell if the build has failed (as long as it got to the stage that it appears on the CI dashboard): we have a slack channel! Check the channel #ci_alerts on the microboone slack. It will send an alert if the build fails or succeeds, and another alert when the validation plots are available (including a direct link to the webpage where you can see those plots)

If the CI build has failed while running, see this page for instructions about how to use the CI output to find out where your jobs failed.