Automated Validation » History » Version 24
Julia Yarba, 02/25/2019 05:19 PM
1 | 7 | Julia Yarba | h1. GENIE AVS CI - Automated Validation |
---|---|---|---|
2 | 1 | Gabriel Perdue | |
3 | 3 | Julia Yarba | h2. General Information |
4 | 3 | Julia Yarba | |
5 | 3 | Julia Yarba | GENIE AVS CI is GENIE Automated Validation Suite combined with the "Continuous Integration (CI) services":https://cdcvs.fnal.gov/redmine/projects/ci. |
6 | 3 | Julia Yarba | |
7 | 21 | Julia Yarba | The purpose of GENIE Validation Suite is to monitor, throughout the development process, the state of the GENIE core code and the evolution of the GENIE physics models, through the benchmark of the simulated results vs selected experimental data. |
8 | 21 | Julia Yarba | |
9 | 21 | Julia Yarba | The same procedure and set of software tools is used to validate public GENIE releases (the latest release is R-3_00_02). |
10 | 21 | Julia Yarba | |
11 | 21 | Julia Yarba | It is also being considered how to share with the users community appropriate results from GENIE benchmark vs experimental experimental data, for public GENIE releases. |
12 | 22 | Julia Yarba | Currently we offer, through "DoSSiER":https://g4devel.fnal.gov:8181/DoSSiER repository, a small prototype that illustrate the "benchmark of GENIE simulated results vs selected MINERvA data.":https://g4devel.fnal.gov:8181/DoSSiER/SelectGENIETest.xhtml |
13 | 22 | Julia Yarba | Please click on the "100000" identifier in the left column of the table to go to the results display. |
14 | 21 | Julia Yarba | Included simulated results are from releases R-2_12_10, R-3_00_00, and R-3_00_02, for tunes G00_00a_00_000, G18_02a_00_000, G18_02a_02_11a, and G18_10j_00_000, as applicable. |
15 | 21 | Julia Yarba | |
16 | 23 | Julia Yarba | Several tops on navigating DoSSiER/GENIE-MINERvA validation results display are offered later in this document. |
17 | 23 | Julia Yarba | |
18 | 21 | Julia Yarba | h2. Overview of GENIE AVS CI |
19 | 21 | Julia Yarba | |
20 | 3 | Julia Yarba | The procedure can be subdivided into 2 categories: |
21 | 3 | Julia Yarba | * GENIE CI build(s) "on commit" |
22 | 3 | Julia Yarba | ** Triggered every time when new code is committed (to GitHib) to either Generator, Reweighs, or Comparisons package (given 20min wait time, to allow consolidation of multiple commits within a short time frame) |
23 | 3 | Julia Yarba | ** Includes the followjng steps: |
24 | 3 | Julia Yarba | *** Checkout from GitHub |
25 | 3 | Julia Yarba | *** Build of all 3 packages |
26 | 3 | Julia Yarba | *** Execution of Unit Tests |
27 | 8 | Julia Yarba | *** "Operational" test which includes |
28 | 8 | Julia Yarba | a) generation of minimalistic cross section splines for the CCQE process |
29 | 9 | Julia Yarba | b) generation of 5 CCQE events using splines generated in a) |
30 | 8 | Julia Yarba | both are done for the Default tune |
31 | 3 | Julia Yarba | * GENIE Validation workflow is triggered weekly (on Friday's) |
32 | 3 | Julia Yarba | ** GENIE build, Unit Tests, and "operational" test, as described above |
33 | 7 | Julia Yarba | ** Benchmark of GENIE generated predictions vs a variety of experimental datasets; the list of benchmarks can be found "here":https://cdcvs.fnal.gov/redmine/projects/genie_ci/wiki/grid_phase |
34 | 3 | Julia Yarba | |
35 | 5 | Julia Yarba | Results are reported to the dedicated "GENIEMC Slack channel":https://geniemc.slack.com/messages/C50RB57JS/convo/GA63C15T7-1539263877.000100/ (password may be needed) |
36 | 12 | Julia Yarba | Each report includes link(s) to the "GENIE CI dashboard":http://genie-ci-history.fnal.gov/GenieCI/app/view_builds/index for further details (log files, plot books, etc.) |
37 | 5 | Julia Yarba | |
38 | 13 | Julia Yarba | Various technical details can also be found in the presentation at the GENIE meetings; see later in this document. |
39 | 3 | Julia Yarba | |
40 | 17 | Julia Yarba | h2. Some Tips on Interpreting GENIE CI Dashboard |
41 | 14 | Julia Yarba | |
42 | 14 | Julia Yarba | Each raw on the "GENIE CI dashboard":http://genie-ci-history.fnal.gov/GenieCI/app/view_builds/index corresponds to a separate workflow. |
43 | 14 | Julia Yarba | Obviously, the workflows are shorter for the "on commit" builds and longer for the full blown GENIE AVS rounds. |
44 | 14 | Julia Yarba | |
45 | 14 | Julia Yarba | On the dashboard this is reflected wit the addition of the following: |
46 | 14 | Julia Yarba | * "setup_validation" section - preparation of the storage area in dCache, generation of job configuration files, etc. |
47 | 14 | Julia Yarba | * one or several "ci_validation_<TuneID> sections (e.g. ci_validation_G18_02a_00_000, etc.) |
48 | 14 | Julia Yarba | |
49 | 16 | Julia Yarba | Each ci_validation_<TuneID> section is a folder on its own. |
50 | 14 | Julia Yarba | If one clicks on it, one will be redirected to another monitoring board that corresponds to the validation of a given GENIE tune. |
51 | 14 | Julia Yarba | Such validation procedure consists of multiple steps and includes generation of neutrino-nucleon, then neutrino-nucleus splines, generation of GENIE event files for various MC-data benchmarks, |
52 | 14 | Julia Yarba | and the MC-data benchmarks themselves. |
53 | 14 | Julia Yarba | They are named as dag_<NN>, where NN is the serial number of a particular step in the validation chain. |
54 | 15 | Julia Yarba | The notation of "dag" is associated with the term DAGman (it is a meta-scheduler for HTCondor that menages dependencies between jobs at a high level). |
55 | 14 | Julia Yarba | |
56 | 14 | Julia Yarba | Description of what each dag_<NN> actually means/does is linked to the "?" icon in the box on the left side of the web page, located below the "Progress Legend" chart; feel free to click on the icon. |
57 | 14 | Julia Yarba | |
58 | 24 | Julia Yarba | h2. Some Tips on Navigating DoSSiER/GENIE-MINERvA Display |
59 | 24 | Julia Yarba | |
60 | 24 | Julia Yarba | "DoSSiER/GENIE-MINERvA":https://g4devel.fnal.gov:8181/DoSSiER/SelectGENIETest.xhtml should show the list of publications that the experimental data were taken from, along with the "default" plot of a GENIE-MINERvA benchmark. The "default" plot show results on single pi0 production, as simulated with R-2_12_10, compared vs MINERvA experimental data published in 2015. |
61 | 24 | Julia Yarba | |
62 | 24 | Julia Yarba | One can make other selections through the pull down menu's such as "Beam" (e.g. flux), "Reaction", "Observable", VersionTag", and "Model" (e.g. tune). |
63 | 24 | Julia Yarba | |
64 | 24 | Julia Yarba | As an exercise, one can deselect VersionTag R-2_12_10 and select R-3_00_02 instead, then deselect Model (tune) G00_00a_00_000 and select G18_02a_00_00_000, G18_02a_02_11am and G18_10j_00_000, to see how different tunes benchmark vs each other and vs experimental data, for this particular distribution (dSigma/dtheta ion the secondary pi0). |
65 | 24 | Julia Yarba | |
66 | 24 | Julia Yarba | One can also see other distributions by selecting another Beam (flux), Reaction, and Observable. |
67 | 24 | Julia Yarba | |
68 | 24 | Julia Yarba | Results will be displayed only if a combination of choices is valid (i.e. there're corresponding results). |
69 | 24 | Julia Yarba | E.g. if one chooses tune G00_00a_00_000 and release (VersionTag) R-3_00_02, the output will be empty (in other words, an "educated user" is anticipated). |
70 | 24 | Julia Yarba | |
71 | 24 | Julia Yarba | |
72 | 24 | Julia Yarba | |
73 | 24 | Julia Yarba | |
74 | 3 | Julia Yarba | h2. Presentations on GENIE AVS CI |
75 | 3 | Julia Yarba | |
76 | 6 | Julia Yarba | * "GENIE AVS CO Overview at the STWG meeting on 6/13/2018":https://cdcvs.fnal.gov/redmine/attachments/52114/GENIE-AVS-CI-Status-June2018.pdf |
77 | 3 | Julia Yarba | NOTE: A few details may be slightly out of date as a result of updates implemented since June 2018 |
78 | 3 | Julia Yarba | |
79 | 10 | Julia Yarba | h2. How to Launch GENIE AVS CI |
80 | 3 | Julia Yarba | |
81 | 3 | Julia Yarba | GENIE "on commit" and/or weekly AVS CI are triggered automatically. |
82 | 3 | Julia Yarba | |
83 | 18 | Julia Yarba | However, the procedure can be triggered manually at any given moment, if needs be. |
84 | 3 | Julia Yarba | |
85 | 20 | Julia Yarba | Any authorized GENIE developer can trigger one or another workflow, following "these instructions":https://cdcvs.fnal.gov/redmine/projects/genie_ci/wiki |
86 | 1 | Gabriel Perdue | |
87 | 19 | Julia Yarba | h2. Legacy Materials |
88 | 2 | Julia Yarba | |
89 | 2 | Julia Yarba | * Meetings |
90 | 2 | Julia Yarba | *** [[AVS - April 2, 2015]] |