Project

General

Profile

Wiki » History » Version 1

Michele Fattoruso, 03/01/2017 02:11 PM

1 1 Michele Fattoruso
h1. How to setup and trigger your CI build
2 1 Michele Fattoruso
3 1 Michele Fattoruso
{{>toc}}
4 1 Michele Fattoruso
5 1 Michele Fattoruso
p=. %{color:red} NOTE: THE WIKI IS WORK IN PROGRESS%
6 1 Michele Fattoruso
%{color:red} the information are correct but the images are for information purpose only. the images aren't related to this project but are a good example of how the system will look,some command are still related with another experiment but they will be updated soon%
7 1 Michele Fattoruso
8 1 Michele Fattoruso
h2. How to setup the environment
9 1 Michele Fattoruso
10 1 Michele Fattoruso
11 1 Michele Fattoruso
To properly work on the CI system you first need to download and/or setup the code from the repository
12 1 Michele Fattoruso
13 1 Michele Fattoruso
%{color:red} NOTE: Before  executing the code, move into the directory where you want to keep the code%
14 1 Michele Fattoruso
15 1 Michele Fattoruso
<pre>
16 1 Michele Fattoruso
source /grid/fermiapp/products/setups.sh
17 1 Michele Fattoruso
kinit
18 1 Michele Fattoruso
setup cigetcert
19 1 Michele Fattoruso
cigetcert -s fifebatch.fnal.gov
20 1 Michele Fattoruso
#voms-proxy-init -noregen -rfc -voms fermilab:/fermilab/uboone/Role=Analysis
21 1 Michele Fattoruso
22 1 Michele Fattoruso
mkdir my_ci_dir
23 1 Michele Fattoruso
cd my_ci_dir
24 1 Michele Fattoruso
25 1 Michele Fattoruso
git clone http://cdcvs.fnal.gov/projects/generic_ci
26 1 Michele Fattoruso
git clone http://cdcvs.fnal.gov/projects/lar_ci
27 1 Michele Fattoruso
28 1 Michele Fattoruso
cd generic_ci
29 1 Michele Fattoruso
setup -. generic_ci
30 1 Michele Fattoruso
PATH="${PATH}:${PWD}/bin"
31 1 Michele Fattoruso
32 1 Michele Fattoruso
cd ../lar_ci
33 1 Michele Fattoruso
setup -. -j lar_ci
34 1 Michele Fattoruso
35 1 Michele Fattoruso
</pre>
36 1 Michele Fattoruso
37 1 Michele Fattoruso
The previous code allows you to setup a voms proxy with Analysis role, needed to access files required to run CI tests from the dCache directory.
38 1 Michele Fattoruso
Then you setup generic_ci UPS product then checkout and setup the lar_ci.
39 1 Michele Fattoruso
40 1 Michele Fattoruso
h2. Tips for a faster setup
41 1 Michele Fattoruso
42 1 Michele Fattoruso
The CI environment needs to be configured every time you log into the machine.
43 1 Michele Fattoruso
If you haven't deleted the folder the first time you configured the CI System, you can avoid to download again the code from the repository and use the same code that is on the machine.
44 1 Michele Fattoruso
If you plan to use the local build more than once, could be good to setup a function into your .bashrc file.
45 1 Michele Fattoruso
To do that execute the following steps:
46 1 Michele Fattoruso
47 1 Michele Fattoruso
%{color:blue}NOTE: I'm using vim as text editor but you can open the file with your favorite text editor.%
48 1 Michele Fattoruso
49 1 Michele Fattoruso
# Open the .bashrc file with a text editor
50 1 Michele Fattoruso
<pre>vim ~/.bashrc</pre>
51 1 Michele Fattoruso
# Paste the following code to the end of the file: %{color:blue}P.S: In vim you need to press "i" to activate the insert mode%
52 1 Michele Fattoruso
<pre>
53 1 Michele Fattoruso
function setup_ci
54 1 Michele Fattoruso
{
55 1 Michele Fattoruso
    my_path="${1:-${PWD}}"
56 1 Michele Fattoruso
    
57 1 Michele Fattoruso
    source /grid/fermiapp/products/setups.sh
58 1 Michele Fattoruso
    kinit
59 1 Michele Fattoruso
    setup cigetcert
60 1 Michele Fattoruso
    cigetcert -s fifebatch.fnal.gov
61 1 Michele Fattoruso
    voms-proxy-init -noregen -rfc -voms fermilab:/fermilab/nova/Role=Analysis
62 1 Michele Fattoruso
    
63 1 Michele Fattoruso
    setup generic_ci
64 1 Michele Fattoruso
65 1 Michele Fattoruso
    cd ${my_path}/lar_ci
66 1 Michele Fattoruso
    setup -. -j lar_ci 
67 1 Michele Fattoruso
}
68 1 Michele Fattoruso
</pre>
69 1 Michele Fattoruso
# Save the changes and close the file.
70 1 Michele Fattoruso
%{color:blue}NOTE: In vim you need to press Esc then type ":wq" and then press Enter to save and exit%
71 1 Michele Fattoruso
72 1 Michele Fattoruso
Once you completed the previous three steps, every time you log into the machine you can setup the CI Environment using one of the following methods:
73 1 Michele Fattoruso
# giving as argument to the function the full path of the my_ci_work_dir directory:
74 1 Michele Fattoruso
<pre>setup_ci <path/of/your/my_ci_work_dir/directory></pre>
75 1 Michele Fattoruso
# Running the function without argument if you first change directory to the my_ci_work_dir directory
76 1 Michele Fattoruso
<pre>
77 1 Michele Fattoruso
cd <path/of/your/my_ci_work_dir/directory>
78 1 Michele Fattoruso
​setup_ci
79 1 Michele Fattoruso
</pre>
80 1 Michele Fattoruso
81 1 Michele Fattoruso
h2. Manually trigger a build
82 1 Michele Fattoruso
83 1 Michele Fattoruso
Once the environment is set, you can launch the build/workflow executing the trigger script with different options:
84 1 Michele Fattoruso
# *To trigger the default CI build*
85 1 Michele Fattoruso
<pre>trigger</pre>
86 1 Michele Fattoruso
# *To trigger a CI build with a specific tag*
87 1 Michele Fattoruso
<pre>trigger --revisions "novaart.pkgs.svn@S16-03-04"</pre>
88 1 Michele Fattoruso
89 1 Michele Fattoruso
If you want to trigger your build with mrb or SRT there are two options:
90 1 Michele Fattoruso
91 1 Michele Fattoruso
# To build with mrb,use the option "--workflow NOvA_CI_MRB"
92 1 Michele Fattoruso
<pre>trigger --workflow NOvA_CI_MRB</pre>
93 1 Michele Fattoruso
# To build with SRT,use the option "--workflow NOvA_CI_SRT"
94 1 Michele Fattoruso
<pre>trigger --workflow NOvA_CI_SRT</pre> 
95 1 Michele Fattoruso
96 1 Michele Fattoruso
%{color:blue}NOTE: As default, trigger use NOvA_CI_SRT as workflow%
97 1 Michele Fattoruso
98 1 Michele Fattoruso
An email report is sent also to whom manually triggered the CI build, other that to the other mailing list configured in the workflow.cfg.
99 1 Michele Fattoruso
100 1 Michele Fattoruso
For the online help page of trigger, use the following command:
101 1 Michele Fattoruso
102 1 Michele Fattoruso
<pre>trigger -h</pre>
103 1 Michele Fattoruso
104 1 Michele Fattoruso
h3. Manual trigger to generate a new set of reference files
105 1 Michele Fattoruso
106 1 Michele Fattoruso
When you have setup the CI environment you can trigger a manual CI build to generate a new set of reference files for a specific list of CI tests.
107 1 Michele Fattoruso
108 1 Michele Fattoruso
<pre>
109 1 Michele Fattoruso
trigger --build-delay 0 --ci-tests "list of CI tests that requires new reference files" --workflow NOvA_CI_SRT_GEN_REF_FILES 
110 1 Michele Fattoruso
</pre>
111 1 Michele Fattoruso
112 1 Michele Fattoruso
where the "list of CI tests that requires new reference files" uses withe space as separator and should be included within double quotes. If the option is missed the "default" CI test suite will be used.
113 1 Michele Fattoruso
114 1 Michele Fattoruso
The pattern of the reference files is: <filename_prefix>_Reference-<platform>.<filename_suffix>.root
115 1 Michele Fattoruso
The current <platform> is slf6 and it depends on the node where the CI build runs.
116 1 Michele Fattoruso
117 1 Michele Fattoruso
When generating a new set of reference files, the system is forced to process only 1 event.
118 1 Michele Fattoruso
If the experiment code fails to generate the output file, the CI phase is declared *Failed* and the reference files are not uploaded in the experiment dCache area.
119 1 Michele Fattoruso
 
120 1 Michele Fattoruso
%{color:blue}NOTE: If the experiment code fails to generate the output file, CI phase is declared *Failed* and the reference files are not uploaded in the experiment dCache area.
121 1 Michele Fattoruso
So if there are CI tests known to fail, take the failing test out of the CI tests list. You can select also a single CI test for which generate new reference files.
122 1 Michele Fattoruso
This feature allows you to update reference files also if some of the CI test is failing, and only for CI tests that you really need to update references files.
123 1 Michele Fattoruso
%
124 1 Michele Fattoruso
125 1 Michele Fattoruso
The procedure to generate the new set of reference files copies the new reference files in the related STAGE_NAME folder in dCache in the form <filename_prefix>_Reference-<platform>-<timestamp>.<filename_suffix>.root
126 1 Michele Fattoruso
These files are tested using the CI tests, if the CI tests pass, the new reference files are copied as <filename_prefix>_Reference-<platform>.<filename_suffix>.root keeping a copy of the new reference file with the time stamp.
127 1 Michele Fattoruso
The previous reference files are copied in backup files, just in case the new reference file has some issue.
128 1 Michele Fattoruso
The existing backup file is removed.
129 1 Michele Fattoruso
130 1 Michele Fattoruso
h2. Run the CI Tests on the local machine
131 1 Michele Fattoruso
132 1 Michele Fattoruso
%{color:red}NOTE: Before to execute the tests locally move to a clean or at least not important folder, considering that running CI tests you get a new folder for each CI test you are running. Also make sure to have access to enough disk space, depending on the nature of the CI tests executed, you could need few Gb of disk space.%
133 1 Michele Fattoruso
134 1 Michele Fattoruso
To run the CI Tests on the local machine you need to set up the NOvA environment for your local installed NOvA code and the CI environment.
135 1 Michele Fattoruso
136 1 Michele Fattoruso
Once the environment is configured, you can see the list of CI Tests and the list of suites available with the following command:
137 1 Michele Fattoruso
138 1 Michele Fattoruso
<pre>
139 1 Michele Fattoruso
test_runner -l
140 1 Michele Fattoruso
</pre>
141 1 Michele Fattoruso
142 1 Michele Fattoruso
You can then run the CI tests on the local machine with the following command:
143 1 Michele Fattoruso
144 1 Michele Fattoruso
<pre>test_runner <name of the test suite or name of the CI test></pre>
145 1 Michele Fattoruso
146 1 Michele Fattoruso
for test_runner you can use also the following common options:
147 1 Michele Fattoruso
* -v for the verbose mode
148 1 Michele Fattoruso
* -d for the debug mode
149 1 Michele Fattoruso
* -s to get the statistics of the different tests 
150 1 Michele Fattoruso
151 1 Michele Fattoruso
so the command will be:
152 1 Michele Fattoruso
153 1 Michele Fattoruso
<pre>test_runner -v -d -s <name of the test suite or name of the CI test></pre>
154 1 Michele Fattoruso
155 1 Michele Fattoruso
As default test_runner process up to *5* tests in parallel, this value can be modified using the option --parallel-limit.
156 1 Michele Fattoruso
157 1 Michele Fattoruso
For the online help page of the test_runner, use the following command:
158 1 Michele Fattoruso
159 1 Michele Fattoruso
<pre>test_runner -h</pre>
160 1 Michele Fattoruso
161 1 Michele Fattoruso
h1. How to monitor the status of your build
162 1 Michele Fattoruso
163 1 Michele Fattoruso
h2. The Nova_CI Web Application
164 1 Michele Fattoruso
165 1 Michele Fattoruso
To monitor the status of your builds into the CI system, our team created an experiment based web application to help you with your monitoring job.
166 1 Michele Fattoruso
167 1 Michele Fattoruso
The application can be found at the following link:
168 1 Michele Fattoruso
169 1 Michele Fattoruso
http://nova-ci-history.fnal.gov:8080/NovaCI/app/view_builds/index
170 1 Michele Fattoruso
171 1 Michele Fattoruso
This application offer to you all the necessary tools to monitor the status and the performance of your builds in a short and long term.
172 1 Michele Fattoruso
It offers a quick overview of the status of the last ten builds and offers also dynamic graphs to show the overtime performance of the different build phases.
173 1 Michele Fattoruso
174 1 Michele Fattoruso
h2. The main page of the web application
175 1 Michele Fattoruso
176 1 Michele Fattoruso
p=. !{width:90%}Nova_CI_Homepage.png(CI System Homepage)!
177 1 Michele Fattoruso
178 1 Michele Fattoruso
The previous image shows the homepage of the NovaCI web application.
179 1 Michele Fattoruso
180 1 Michele Fattoruso
This page is divided into three main sections:
181 1 Michele Fattoruso
# The search and filter section ( on the left of the image )
182 1 Michele Fattoruso
# The legend section ( on the right of the image )
183 1 Michele Fattoruso
# The Show data section (in the middle of the image )
184 1 Michele Fattoruso
185 1 Michele Fattoruso
h2. The search and filter section
186 1 Michele Fattoruso
187 1 Michele Fattoruso
The purpose of this section is to offer to the user a tool to filter the results shown in the web application.
188 1 Michele Fattoruso
This panel allows you to:
189 1 Michele Fattoruso
* Show the builds up to a defined build number.
190 1 Michele Fattoruso
* Show the builds in a defined time range.
191 1 Michele Fattoruso
* Filter the builds for platform or for test suite used.
192 1 Michele Fattoruso
193 1 Michele Fattoruso
h2. The legend section
194 1 Michele Fattoruso
195 1 Michele Fattoruso
This section show the association between the color of the different phases of the builds,and their status
196 1 Michele Fattoruso
* %{color:yellow} [#]% yellow         -> The phase is running
197 1 Michele Fattoruso
* %{color:gray} [#]% gray            -> The phase is pending
198 1 Michele Fattoruso
*  %{color:green} [#]% green          -> The phase was successful
199 1 Michele Fattoruso
*  %{color:orange} [#]% orange          -> The phase failed in warning status
200 1 Michele Fattoruso
*  %{color:red} [#]% red              -> The phase failed
201 1 Michele Fattoruso
*  %{color:lightblue} [#]% light blue -> The phase have been skipped
202 1 Michele Fattoruso
%{color:blue} NOTE: A warning status is when the tests are successful but the product size or names are different when the output is compared to a reference file%
203 1 Michele Fattoruso
204 1 Michele Fattoruso
h2. The Show data section
205 1 Michele Fattoruso
206 1 Michele Fattoruso
This is the main section of the homepage and it's where the most important data is shown.
207 1 Michele Fattoruso
This section shows a table of the last ten builds executed and their corresponding information.
208 1 Michele Fattoruso
209 1 Michele Fattoruso
The table is divided into multiple columns.
210 1 Michele Fattoruso
211 1 Michele Fattoruso
The first three columns show some basic information about the CI build number, the time when the CI build started and the platform on which the CI build have been executed.
212 1 Michele Fattoruso
213 1 Michele Fattoruso
The next columns show the status of the different phases of the CI build.
214 1 Michele Fattoruso
Each phase is represented by a square with a bullet
215 1 Michele Fattoruso
216 1 Michele Fattoruso
In this moment a Nova CI build with the NOVA_CI_SRT workflow is divided into three phases:
217 1 Michele Fattoruso
# Checkout
218 1 Michele Fattoruso
Show the status of the checkout of the code from the repository.
219 1 Michele Fattoruso
# Build
220 1 Michele Fattoruso
shows the status of the building process of the code into the CI System.
221 1 Michele Fattoruso
# CI tests
222 1 Michele Fattoruso
Show the progress of the CI tests defined for this build.
223 1 Michele Fattoruso
224 1 Michele Fattoruso
A Nova CI build with the NOVA_CI_MRB workflow have an extra phase:
225 1 Michele Fattoruso
# Install
226 1 Michele Fattoruso
Show the progress of the install process of the code into the CI System.
227 1 Michele Fattoruso
228 1 Michele Fattoruso
h1. Dynamic graphs and statistics
229 1 Michele Fattoruso
230 1 Michele Fattoruso
Every bullet and the build number into the web application is clickable.
231 1 Michele Fattoruso
Performing a click on a bullet open a different web page that shows detailed information on the phase.
232 1 Michele Fattoruso
233 1 Michele Fattoruso
h2. [[Build Details]]
234 1 Michele Fattoruso
235 1 Michele Fattoruso
h2. [[Checkout Phase Details]]
236 1 Michele Fattoruso
237 1 Michele Fattoruso
h2. [[Build Phase Details]]
238 1 Michele Fattoruso
239 1 Michele Fattoruso
h2. [[Unit Tests Details]]
240 1 Michele Fattoruso
241 1 Michele Fattoruso
h2. [[Install Phase Details]]
242 1 Michele Fattoruso
243 1 Michele Fattoruso
h2. [[CI Tests Phase Details]]