Automated Validation » History » Version 22

Julia Yarba, 02/25/2019 04:04 PM

1 7 Julia Yarba
h1. GENIE AVS CI - Automated Validation
2 1 Gabriel Perdue
3 3 Julia Yarba
h2. General Information
4 3 Julia Yarba
5 3 Julia Yarba
GENIE AVS CI is GENIE Automated Validation Suite combined with the "Continuous Integration (CI) services":
6 3 Julia Yarba
7 21 Julia Yarba
The purpose of GENIE Validation Suite is to monitor, throughout the development process, the state of the GENIE core code and the evolution of the GENIE physics models, through the benchmark of the simulated results vs selected experimental data. 
8 21 Julia Yarba
9 21 Julia Yarba
The same procedure and set of software tools is used to validate public GENIE releases (the latest release is R-3_00_02).
10 21 Julia Yarba
11 21 Julia Yarba
It is also being considered how to share with the users community appropriate results from GENIE benchmark vs experimental experimental data, for public GENIE releases.
12 22 Julia Yarba
Currently we offer, through "DoSSiER": repository, a small prototype that illustrate the "benchmark of GENIE simulated results vs selected MINERvA data.":
13 22 Julia Yarba
Please click on the "100000" identifier in the left column of the table to go to the results display.
14 21 Julia Yarba
Included simulated results are from releases R-2_12_10, R-3_00_00, and R-3_00_02, for tunes G00_00a_00_000, G18_02a_00_000, G18_02a_02_11a, and G18_10j_00_000, as applicable.
15 21 Julia Yarba
16 21 Julia Yarba
h2. Overview of GENIE AVS CI
17 21 Julia Yarba
18 3 Julia Yarba
The procedure can be subdivided into 2 categories:
19 3 Julia Yarba
* GENIE CI build(s) "on commit"
20 3 Julia Yarba
** Triggered every time when new code is committed (to GitHib) to either Generator, Reweighs, or Comparisons package (given 20min wait time, to allow consolidation of multiple commits within a short time frame)
21 3 Julia Yarba
** Includes the followjng steps:
22 3 Julia Yarba
*** Checkout from GitHub
23 3 Julia Yarba
*** Build of all 3 packages
24 3 Julia Yarba
*** Execution of Unit Tests
25 8 Julia Yarba
*** "Operational" test which includes 
26 8 Julia Yarba
a) generation of minimalistic cross section splines for the CCQE process 
27 9 Julia Yarba
b) generation of 5 CCQE events using splines generated in a) 
28 8 Julia Yarba
both are done for the Default tune   
29 3 Julia Yarba
* GENIE Validation workflow is triggered weekly (on Friday's)
30 3 Julia Yarba
** GENIE build, Unit Tests, and "operational" test,  as described above
31 7 Julia Yarba
** Benchmark of GENIE generated predictions vs a variety of experimental datasets; the list of benchmarks can be found "here":
32 3 Julia Yarba
33 5 Julia Yarba
Results are reported to the dedicated "GENIEMC Slack channel": (password may be needed)
34 12 Julia Yarba
Each report includes link(s) to the "GENIE CI dashboard": for further details (log files, plot books, etc.)
35 5 Julia Yarba
36 13 Julia Yarba
Various technical details can also be found in the presentation at the GENIE meetings; see later in this document.
37 3 Julia Yarba
38 17 Julia Yarba
h2. Some Tips on Interpreting GENIE CI Dashboard
39 14 Julia Yarba
40 14 Julia Yarba
Each raw on the "GENIE CI dashboard": corresponds to a separate workflow.
41 14 Julia Yarba
Obviously, the workflows are shorter for the "on commit" builds and longer for the full blown GENIE AVS rounds.
42 14 Julia Yarba
43 14 Julia Yarba
On the dashboard this is reflected wit the addition of the following: 
44 14 Julia Yarba
* "setup_validation" section - preparation of the storage area in dCache, generation of job configuration files, etc. 
45 14 Julia Yarba
* one or several "ci_validation_<TuneID> sections (e.g. ci_validation_G18_02a_00_000, etc.)
46 14 Julia Yarba
47 16 Julia Yarba
Each ci_validation_<TuneID> section is a folder on its own.
48 14 Julia Yarba
If one clicks on it, one will be redirected to another monitoring board that corresponds to the validation of a given GENIE tune.
49 14 Julia Yarba
Such validation procedure consists of multiple steps and includes generation of neutrino-nucleon, then neutrino-nucleus splines, generation of GENIE event files for various MC-data benchmarks, 
50 14 Julia Yarba
and the MC-data benchmarks themselves.
51 14 Julia Yarba
They are named as dag_<NN>, where NN is the serial number of a particular step in the validation chain.
52 15 Julia Yarba
The notation of "dag" is associated with the term DAGman (it is a meta-scheduler for HTCondor that menages dependencies between jobs at a high level).
53 14 Julia Yarba
54 14 Julia Yarba
Description of what each dag_<NN> actually means/does is linked to the "?" icon in the box on the left side of the web page, located below the "Progress Legend" chart; feel free to click on the icon.
55 14 Julia Yarba
56 3 Julia Yarba
h2. Presentations on GENIE AVS CI
57 3 Julia Yarba
58 6 Julia Yarba
* "GENIE AVS CO Overview at the STWG meeting on 6/13/2018":
59 3 Julia Yarba
NOTE: A few details may be slightly out of date as a result of updates implemented since June 2018
60 3 Julia Yarba
61 10 Julia Yarba
h2. How to Launch GENIE AVS CI
62 3 Julia Yarba
63 3 Julia Yarba
GENIE "on commit" and/or weekly AVS CI are triggered automatically.
64 3 Julia Yarba
65 18 Julia Yarba
However, the procedure can be triggered manually at any given moment, if needs be.
66 3 Julia Yarba
67 20 Julia Yarba
Any authorized GENIE developer can trigger one or another workflow, following "these instructions":
68 1 Gabriel Perdue
69 19 Julia Yarba
h2. Legacy Materials
70 2 Julia Yarba
71 2 Julia Yarba
* Meetings
72 2 Julia Yarba
*** [[AVS - April 2, 2015]]