CI Validation Instructions¶
- Table of contents
- CI Validation Instructions
First of all, thank you for helping with the uboonecode validation! CI validation is vital to ensure we catch any bugs as soon as possible. As a shifter you are playing a vital role! Below you will see instructions to help you access the validation plots, determine if they show any problems, and work out what action items to take.
How to contact the experts¶
You can find the list of people to contact in the "CI validation" project (on the elog, click on "Projects" in the top navigation bar, then click on "CI validation" in the list).
Ahh! What is this?¶
CI stands for "Continuous Integration" validation, or validation of every tag of uboonecode - not just when we intend to cut a release. Every week when the production branch uboonecode is tagged 10,000 events are generated and processed through detsim, reco1, reco2, and anatree, and a selection of validation plots are produced. These are split into two categories:
- Reco-level comparisons using cosmic data and Corsika MC
- Truth- and reco-level comparisons using BNB MC (these allow us to see the absolute performance of the reconstruction in terms of e.g. vertex resolution)
For each of these categories we produce two sets of comparisons:
- This week to last week ("prevMC")
- This week to a base release ("refMC"). The current base release is MCC 8.4 (uboonecode v06_26_01_05). You do not need to check these plots as a shifter - your responsibility is just to check the "prevMC" plots.
Your responsibility as shifter is to check the "prevMC" comparison plots between this week and last week (on the first shift of your block only) and inform the experts if you notice anything strange so it can be quickly addressed. Note that because the events are generated from scratch each week, we expect to see statistical fluctuations. What you are looking for are differences that don't seem to be consistent with statistical fluctuations.
Quickstart: what you need to do¶
This is a quick guide to using the CI validation dashboard. For more detailed explanations, see more in-depth instructions here.
- Open the CI validation plots:
- Follow this link for the reco-level comparisons using cosmic MC
- Follow this link for the reco-level comparisons using cosmic data
- Follow this link for the reco- and truth-level comparisons using BNB MC
- If prompted for a username and password, you should input your Fermilab Services account username and password
- After following each link, navigate to the most recent CI validation plots by clicking on the highest box in the right column (labelled "ci_validation") with a green check or orange warning sign in it. If the box contains a yellow circle and says "pending" when you hover the mouse over it, try the next row down.
- For more information about how to navigate these pages, see "How to get to the validation plots" further down on this page.
- Check the plots. Only look at the comparisons between this week's release and last week's release ("compare_MC_prevMC"). Look at these comparisons for both the reco-level cosmic data/MC and the reco/truth-level BNB MC.
- Complete the CI Validation checklist in the ECL based on what you see in the plots (click "New Entry" on the left, then choose "CI Validation Checklist" in the drop-down box next to "Form").
- If you see any unexpected features in the plots, see instructions on actions to take.
What should I do if I see a problem?¶
If you see something strange in the plots, there are three methods at your disposal to check whether this is a new difference that has not already been recognized:
- Check the What Level Of Statistical Fluctuations Do I Expect? page, which documents the level of statistical fluctuations you should expect to see, comparing two statistically independent sets of 10,000 events generated from the same tag of uboonecode.
- Also check this page: Log of known changes in uboonecode and effect on validation plots. This page is under development. It is intended to document the current known differences between the develop and production branches, and to be regularly updated to reflect differences that shifters have noticed (when they have been understood).
If the problem that you see is not already listed on these pages, please note it in your CI validation checklist elog entry, preferably including screenshots of the plots in question if possible (click "New Entry" on the left, then choose "CI Validation Checklist" in the drop-down box next to "Form") and email the experts to inform us!
For reference: there is also an Example plots - Comparison to base MC page, documenting example plots from comparison of develop (v06_66_00) to MCC 8.4 (v06_26_01_05). You do not need to check this page because it is not a shifter responsibility to validate comparisons to MCC 8.4 - you only need to validate the comparisons to last week's release ("compareMC_prevMC").
How to get to the validation plots¶
Cosmic reco-level comparisons¶
Look for the column on the right with heading "ci_validation". You may need to scroll to the right or maximize your window. Check the status symbol in the top right box (this is the most recent validation release)
If the status symbol shows a yellow circle (for "pending") then the plots are not ready to look at yet.
If the status symbol shows a green tick/check or an orange triangle, then you should be able to look at the plots.
Click on the most recent (closest to the top) box in this column with a green tick or orange triangle to access the plots page. It will look like this:
There are two things of interest in the above picture.
1. Inside the red box are two links:
- "compare_MC_prevMC" - comparisons of the current release to the previous tag
- "compare_MC_refMC" - comparisons of the current release to MCC 8.6 (uboonecode v06_26_01_10) under "compare_MC_refMC")
Click on "compare_MC_prevMC" to expand the page and see the plots. These are the plots that we'd like you to check! You do not need to check "compare_MC_refMC".
2. This is mostly for experts, but left here for particularly curious shifters. If you see anything strange in any of these plots, you can follow the links in the blue box to track how chi2 has changed over time. This will help you work out if what you see is a recent change or has been known about for some time. For example, clicking on "compare_MC_prevMC/flsPe_simpleFlashBeam" will show you the chi2 for the flsPe_simpleFlashBeam plot between the each release and the previous tag. The history for this only goes back as far as we've been performing CI validation, and so long as that plot has been added. New plots won't have a long chi-squared history.
BNB reco-/truth-level comparisons¶
Follow this link to get to the CI dashboard page for the BNB reco- and truth-level comparisons. It should look similar to the page for cosmic validations. Navigate to the latest CI validation plots in the same way (scroll to the right to see the column "ci_validation" and click on the highest box that has a green check or an orange warning sign).
Once you get to the validation page, it will look almost exactly the same as for the cosmic validations.
CI validation: further information for developers¶Follow these links for more information about:
- Extra tips and tricks for navigating the CI validation dashboard
- Triggering the CI validation
- How to cancel a CI validation
- Checking whether the CI validation has worked
- Finding log files and debugging information if the CI build fails
- Finding output files from the CI validation for further tests
- How to run the CI validation scripts locally
- Full documentation of CI validation scripts: Docdb-13373
- Documentation of CI validation: Docdb-13373