Project

General

Profile

Notes for NuTools release managers » History » Version 47

Lynn Garren, 03/03/2020 11:24 AM

1 1 Lynn Garren
h1. Notes for NuTools release managers
2 1 Lynn Garren
3 4 Lynn Garren
{{>TOC}}
4 4 Lynn Garren
5 6 Lynn Garren
h2(count). Updating one package
6 1 Lynn Garren
7 1 Lynn Garren
* If you only need a new version of a single package, such as nugen, then simply checkout that package, edit, tag and build.
8 1 Lynn Garren
9 5 Lynn Garren
* If you want a complete nulite distribution with the new package, then you would also checkout nutools and follow the instructions for updating the nu suite.
10 2 Lynn Garren
11 2 Lynn Garren
h2(count). The nu suite
12 2 Lynn Garren
13 2 Lynn Garren
The nu suite consists of nutools, nuevdb, nug4, nugen, nurandom, and nusimdata.
14 2 Lynn Garren
15 26 Lynn Garren
16 26 Lynn Garren
h3(count). Branch names
17 26 Lynn Garren
18 26 Lynn Garren
* Branches for NOvA are of the form nova_v3_01_br
19 26 Lynn Garren
* Branches for LArSoft are of the form lar_v2_27_br
20 26 Lynn Garren
* Because LArSoft moves ahead with art more frequently than NOvA, most larsoft distributions are made off the head of develop.
21 26 Lynn Garren
22 2 Lynn Garren
h2(count).  Tools
23 2 Lynn Garren
24 19 Lynn Garren
* mrb v3_04_04 or later 
25 19 Lynn Garren
** This version adds the nu_suite gitCheckout option and fixes some bugs in mrb uv.
26 2 Lynn Garren
* buildFW (used by the Jenkins build)
27 7 Lynn Garren
* copyFromJenkins
28 7 Lynn Garren
* copyToSciSoft
29 8 Lynn Garren
* pullProducts
30 25 Lynn Garren
* pullPackage
31 44 Lynn Garren
* larreltools v1_08_04 or later
32 25 Lynn Garren
** This package was created for LArSoft, but contains common tools.
33 1 Lynn Garren
** dogit
34 5 Lynn Garren
** startNuLiteRel
35 35 Lynn Garren
** manageNuLite
36 1 Lynn Garren
37 27 Lynn Garren
h2(count). Updating the entire nutools suite from the head of develop
38 9 Lynn Garren
39 26 Lynn Garren
These instructions presume that the work is done on scisoftbuild01.fnal.gov.  These instructions also presume that the head of develop will be used. 
40 10 Lynn Garren
41 14 Lynn Garren
h3(count).  update "third party" products
42 14 Lynn Garren
43 15 Lynn Garren
Before building the nu suite, new releases of "third party" products need to be available.  Our definition of "third party products" is any product that does not use cetbuildtools or cetmodules.
44 15 Lynn Garren
45 15 Lynn Garren
This example updates the nu suite to use art 3.04.00.  python3 is the default. This means that the py3 qualifier is no longer used.  Instead, the py2 qualifier is used to specify a build with python v2_7_15a.  Because of this change, root has been updated to v6_18_04c.  Many products in the nulite distribution will need new releases with updated table files.  
46 15 Lynn Garren
47 21 Lynn Garren
* genie v3_00_06e
48 21 Lynn Garren
* dk2nudata v1_08_00f
49 21 Lynn Garren
* dk2nugenie v1_08_00h
50 14 Lynn Garren
51 10 Lynn Garren
h3(count). setup the working environment
52 10 Lynn Garren
53 9 Lynn Garren
<pre>
54 9 Lynn Garren
source /products/setup
55 9 Lynn Garren
setup mrb
56 9 Lynn Garren
setup larreltools
57 9 Lynn Garren
export PATH=$LARRELTOOLS_DIR/expert:${PATH}
58 9 Lynn Garren
export MRB_PROJECT=nu
59 9 Lynn Garren
</pre>
60 9 Lynn Garren
61 1 Lynn Garren
h3(count). make a working directory and checkout the code
62 1 Lynn Garren
63 26 Lynn Garren
Use the startNuLiteRel script.
64 26 Lynn Garren
<pre>
65 26 Lynn Garren
Usage: startNuLiteRel <working_dir> <new tag> <base qualifier>
66 26 Lynn Garren
   This script will do the following:
67 26 Lynn Garren
       mkdir <working_dir>/<tag>
68 26 Lynn Garren
       run newDev
69 26 Lynn Garren
       git clone nulite repositories from redmine
70 26 Lynn Garren
   The base qualifier should be one of e17, e19, c7 etc.
71 26 Lynn Garren
</pre>
72 26 Lynn Garren
For instance:
73 10 Lynn Garren
<pre>
74 25 Lynn Garren
startNuLiteRel `pwd` v3_06_00 e19
75 25 Lynn Garren
startNuLiteRel `pwd` v3_06_00 c7
76 13 Lynn Garren
</pre>
77 13 Lynn Garren
78 13 Lynn Garren
79 12 Lynn Garren
h3(count). update, build, and test the code
80 12 Lynn Garren
81 12 Lynn Garren
Create a temporary release branch.  This workflow mimics, but does not use, gitflow.  This allows us more flexibility when dealing with multiple repositories.
82 12 Lynn Garren
<pre>
83 12 Lynn Garren
source v3_06_00/e19p/localProducts_nu_v3_06_00_e19_prof/setup
84 12 Lynn Garren
cd $MRB_SOURCE
85 30 Lynn Garren
manageNuLite start $MRB_PROJECT_VERSION
86 12 Lynn Garren
</pre>
87 12 Lynn Garren
Now, we update the code.  This example updates the head of the nutools suite to use art 3.04.
88 13 Lynn Garren
<pre>
89 13 Lynn Garren
mrb uv canvas_root_io v1_05_00
90 16 Lynn Garren
mrb uv art v3_04_00
91 13 Lynn Garren
mrb uv art_root_io v1_02_00
92 16 Lynn Garren
mrb uv gallery v1_14_00
93 17 Lynn Garren
mrb uv toyExperiment v0_05_00
94 1 Lynn Garren
mrb uv ifdh_art v2_10_01
95 22 Lynn Garren
</pre>
96 24 Lynn Garren
In this case, we use mrb cq (changeQualifier) from py3 to py2 and also update the version_set.
97 22 Lynn Garren
<pre>
98 22 Lynn Garren
mrb uv version_set s94
99 22 Lynn Garren
mrb cq py3 py2
100 13 Lynn Garren
</pre>
101 30 Lynn Garren
Make sure the required products are available.  We install the critic distribution in /products on scisoftbuild01.  If there are local builds of other products, make sure they are installed in any private product directory.  Add symbolic links to the product in $MRB_INSTALL.  If you change PRODUCTS to point to your product directory, make sure it does not contain test versions of products.  
102 23 Lynn Garren
<pre>
103 23 Lynn Garren
cd $MRB_INSTALL
104 23 Lynn Garren
ln -s /home/garren/scratch/products/genie
105 23 Lynn Garren
etc.
106 23 Lynn Garren
</pre>
107 23 Lynn Garren
Build and test.  Iterate as necessary until everything builds cleanly.
108 23 Lynn Garren
<pre>
109 23 Lynn Garren
cd $MRB_BUILDDIR
110 23 Lynn Garren
mrb z (when necessary)
111 23 Lynn Garren
mrbsetenv
112 23 Lynn Garren
mrb t -jN
113 23 Lynn Garren
</pre>
114 23 Lynn Garren
115 23 Lynn Garren
h3(count). Tagging the release
116 23 Lynn Garren
117 23 Lynn Garren
First update the version.  In this case, the art release and accompanying qualifiers have changed.  The convention is to update the minor version of the packages in the nu suite.
118 23 Lynn Garren
<pre>
119 23 Lynn Garren
updateVersion minor
120 23 Lynn Garren
</pre>
121 1 Lynn Garren
It's wise to have a look at each package and make sure the changes are what you expect.
122 27 Lynn Garren
123 27 Lynn Garren
124 27 Lynn Garren
h2(count). Updating part of the nutools suite.
125 27 Lynn Garren
126 27 Lynn Garren
127 31 Lynn Garren
h3(count).  update required products
128 27 Lynn Garren
129 31 Lynn Garren
Before building the nu suite, new releases of required products need to be available.
130 27 Lynn Garren
131 27 Lynn Garren
This example updates the nu suite to use ifdhc v2_5_4, libwda v2_27_0, and ifdh_art v2_10_02.  
132 27 Lynn Garren
133 27 Lynn Garren
python3 is the default. This means that the py3 qualifier is no longer used.  Instead, the py2 qualifier is used to specify a build with python v2_7_15a.
134 27 Lynn Garren
135 27 Lynn Garren
h3(count). setup the working environment
136 27 Lynn Garren
137 27 Lynn Garren
<pre>
138 27 Lynn Garren
source /products/setup
139 27 Lynn Garren
setup mrb
140 27 Lynn Garren
setup larreltools
141 27 Lynn Garren
export PATH=$LARRELTOOLS_DIR/expert:${PATH}
142 27 Lynn Garren
export MRB_PROJECT=nu
143 27 Lynn Garren
</pre>
144 27 Lynn Garren
145 27 Lynn Garren
h3(count). make a working directory and checkout the code
146 27 Lynn Garren
147 27 Lynn Garren
Use the startNuLiteRel script.
148 27 Lynn Garren
<pre>
149 27 Lynn Garren
Usage: startNuLiteRel <working_dir> <new tag> <base qualifier>
150 27 Lynn Garren
   This script will do the following:
151 27 Lynn Garren
       mkdir <working_dir>/<tag>
152 27 Lynn Garren
       run newDev
153 27 Lynn Garren
       git clone nulite repositories from redmine
154 27 Lynn Garren
   The base qualifier should be one of e17, e19, c7 etc.
155 27 Lynn Garren
</pre>
156 27 Lynn Garren
For instance:
157 27 Lynn Garren
<pre>
158 32 Lynn Garren
startNuLiteRel `pwd` v3_06_00a e19
159 32 Lynn Garren
startNuLiteRel `pwd` v3_06_00a c7
160 27 Lynn Garren
</pre>
161 27 Lynn Garren
162 28 Lynn Garren
h3(count). update the code
163 1 Lynn Garren
164 33 Lynn Garren
We use mrb uv to update all products that are part of the nulite distribution.
165 1 Lynn Garren
<pre>
166 33 Lynn Garren
source v3_06_00a/e19p/local*/setup
167 28 Lynn Garren
cd $MRB_SOURCE
168 1 Lynn Garren
manageNuLite start v3_06_00a
169 1 Lynn Garren
(This step creates a working branch named release/v3_06_00a)
170 33 Lynn Garren
mrb uv ifdhc v2_5_4
171 33 Lynn Garren
mrb uv ifdch_config v2_5_4
172 33 Lynn Garren
mrb uv ifbeam v2_4_2
173 33 Lynn Garren
mrb uv nucondb v2_4_2
174 28 Lynn Garren
mrb uv libwda v2_27_0
175 28 Lynn Garren
mrb uv ifdh_art v2_10_02
176 28 Lynn Garren
</pre>
177 34 Lynn Garren
Check for packages that have no changes and move them out of the way.  This list will vary depending on which dependencies have been updated.
178 34 Lynn Garren
<pre>
179 34 Lynn Garren
cd $MRB_SOURCE
180 34 Lynn Garren
dogit status
181 34 Lynn Garren
mkdir ../nobuild
182 34 Lynn Garren
mv nusimdata ../nobuild/
183 34 Lynn Garren
mv nurandom ../nobuild/
184 34 Lynn Garren
mv nug4 ../nobuild/
185 34 Lynn Garren
mrb uc
186 34 Lynn Garren
</pre>
187 34 Lynn Garren
188 28 Lynn Garren
189 28 Lynn Garren
h3(count). build and test the code
190 23 Lynn Garren
191 29 Lynn Garren
Iterate as necessary until everything builds cleanly.
192 29 Lynn Garren
<pre>
193 29 Lynn Garren
cd $MRB_BUILDDIR
194 29 Lynn Garren
mrb z (if the directory is not empty)
195 29 Lynn Garren
mrbsetenv
196 29 Lynn Garren
mrb t -jN
197 1 Lynn Garren
</pre>
198 1 Lynn Garren
199 29 Lynn Garren
h3(count). update package versions
200 29 Lynn Garren
201 32 Lynn Garren
In this example, we will just update the micro version.  Note that manageNuLite update will update versions in ups/product_deps and in the nutools/bundle directory.  The script automatically calculates the next release number in the appropriate series (major, minor, or micro).
202 29 Lynn Garren
<pre>
203 29 Lynn Garren
manageNuLite update micro
204 29 Lynn Garren
</pre>
205 29 Lynn Garren
206 29 Lynn Garren
h3(count).  special instructions for nutools
207 29 Lynn Garren
208 40 Lynn Garren
nutools contains the build config templates.  These templates must be updated, but there is no reason to make a new release of nutools unless something in nutools itself has changed. If necessary, reset the nutools release to the previous version and update the nulite (distribution) release with an alphanumeric character.  In this example, the only changes in nutools will be to nutools/bundle/CMakeLists.txt.  The nutools/bundle directory contains the distribution templates.
209 29 Lynn Garren
<pre>
210 29 Lynn Garren
cd $MRB_SOURCE/nutools
211 29 Lynn Garren
git diff ups/product_deps
212 1 Lynn Garren
mrb uv nutools v3_06_00
213 1 Lynn Garren
git diff ups/product_deps (should show no changes)
214 1 Lynn Garren
mrb uv nulite v3_06_00a
215 1 Lynn Garren
</pre>
216 31 Lynn Garren
217 31 Lynn Garren
h3(count).  Final build and test
218 31 Lynn Garren
219 31 Lynn Garren
Make one more test build to make sure there are no problems.  This build will also generate the scripts needed to build the nulite distribution.
220 31 Lynn Garren
<pre>
221 31 Lynn Garren
cd $MRB_BUILDDIR
222 31 Lynn Garren
mrb z (if the directory is not empty)
223 31 Lynn Garren
mrbsetenv
224 31 Lynn Garren
mrb t -jN
225 31 Lynn Garren
</pre>
226 31 Lynn Garren
Copy the nulite distribution scripts to SciSoft.
227 31 Lynn Garren
<pre>
228 31 Lynn Garren
cd $MRB_BUILDDIR/nutools/bundle
229 31 Lynn Garren
copyToSciSoft nulite*
230 31 Lynn Garren
</pre>
231 31 Lynn Garren
232 31 Lynn Garren
233 1 Lynn Garren
h3(count). Tag the updated packages
234 1 Lynn Garren
235 35 Lynn Garren
manageNuLite tag will do the following:
236 35 Lynn Garren
* commit changes in ups/product_deps
237 35 Lynn Garren
* merge the release branch with both master and develop
238 35 Lynn Garren
* make the tag on the master branch
239 35 Lynn Garren
* push master, develop, and the tag to the remote repository
240 35 Lynn Garren
* if the nutools release is unchanged, manageNuLite tag will simply merge the release branch with develop and push to origin
241 35 Lynn Garren
* If any file other than ups/product_deps has changed, this step will fail.
242 35 Lynn Garren
* You can rerun this command as many times as necessary without problem.
243 35 Lynn Garren
<pre>
244 35 Lynn Garren
cd $MRB_SOURCE
245 35 Lynn Garren
manageNuLite tag v3_06_00a
246 35 Lynn Garren
</pre>
247 31 Lynn Garren
248 29 Lynn Garren
h2(count). Updating the nutools suite on a branch
249 12 Lynn Garren
250 41 Lynn Garren
* Branches for NOvA are of the form nova_v3_01_br
251 41 Lynn Garren
* Branches for LArSoft are of the form lar_v2_27_br
252 9 Lynn Garren
253 11 Lynn Garren
h3(count). setup the working environment
254 9 Lynn Garren
255 9 Lynn Garren
<pre>
256 9 Lynn Garren
source /cvmfs/nova.opensciencegrid.org/externals/setup
257 9 Lynn Garren
export PRODUCTS=$PRODUCTS:/cvmfs/larsoft.opensciencegrid.org/products
258 9 Lynn Garren
setup mrb
259 9 Lynn Garren
setup larreltools
260 9 Lynn Garren
export PATH=$LARRELTOOLS_DIR/expert:${PATH}
261 9 Lynn Garren
export MRB_PROJECT=nu
262 9 Lynn Garren
</pre>
263 10 Lynn Garren
264 10 Lynn Garren
h3(count). make a working directory and checkout the code
265 10 Lynn Garren
266 10 Lynn Garren
<pre>
267 1 Lynn Garren
makeNuLiteWorkDir `pwd` v3_01_04 e17
268 10 Lynn Garren
makeNuLiteWorkDir `pwd` v3_01_04 c2
269 1 Lynn Garren
</pre>
270 1 Lynn Garren
271 45 Lynn Garren
h2(count). Update nutools for MicroBooNE production releases v08_05_00_NN
272 45 Lynn Garren
273 46 Lynn Garren
The MicroBooNE v08_05_00_NN production releases are built with nutools v2_27_XX.  This was before the nutools v3 split.
274 46 Lynn Garren
275 46 Lynn Garren
Before building nutools, new releases of "third party" products need to be available.
276 46 Lynn Garren
277 46 Lynn Garren
h3(count). setup the working environment
278 46 Lynn Garren
279 46 Lynn Garren
<pre>
280 46 Lynn Garren
source /products/setup
281 46 Lynn Garren
setup mrb
282 46 Lynn Garren
export MRB_PROJECT=nu
283 46 Lynn Garren
</pre>
284 46 Lynn Garren
285 46 Lynn Garren
h3(count). make a working directory and checkout the code
286 46 Lynn Garren
287 46 Lynn Garren
It is not necessary to use mrb.  However, we recommend using mrb so you can update product versions with the "mrb uv" command.
288 46 Lynn Garren
<pre>
289 47 Lynn Garren
mkdir <working_dir>/<new tag>
290 47 Lynn Garren
cd <working_dir>/<new tag>
291 47 Lynn Garren
mrb newDev -v <new tag> -q e17:prof
292 47 Lynn Garren
source <working_dir>/<new tag>/localProducts*/setup
293 46 Lynn Garren
mrb g -b lar_v2_27_br nutools
294 46 Lynn Garren
Verify that you are on the lar_v2_27_br branch of nutools.
295 46 Lynn Garren
</pre>
296 46 Lynn Garren
297 46 Lynn Garren
h3(count). update, build, and test the code
298 46 Lynn Garren
299 46 Lynn Garren
Now, we update the code.  This example updates the head of the nutools suite to use a newer genie and accompanying dk2nudata.
300 46 Lynn Garren
<pre>
301 46 Lynn Garren
mrb uv genie v3_00_04_ub1
302 46 Lynn Garren
mrb uv dk2nugenie v01_08_00_ub1
303 46 Lynn Garren
mrb uv genie_phyopt v3_00_04 (only if needed)
304 46 Lynn Garren
mrb uv genie_xsec v3_00_04a (only if needed)
305 46 Lynn Garren
</pre>
306 46 Lynn Garren
Make sure the required products are available.  If there are local builds of some products, make sure they are installed in a private product directory.  Add symbolic links to the product in $MRB_INSTALL.
307 46 Lynn Garren
<pre>
308 46 Lynn Garren
cd $MRB_INSTALL
309 46 Lynn Garren
ln -s /home/garren/scratch/products/genie
310 46 Lynn Garren
ln -s /home/garren/scratch/products/dk2nugenie
311 46 Lynn Garren
</pre>
312 46 Lynn Garren
Build and test.  Iterate as necessary until everything builds cleanly.
313 46 Lynn Garren
<pre>
314 46 Lynn Garren
cd $MRB_BUILDDIR
315 46 Lynn Garren
mrb z (when necessary)
316 46 Lynn Garren
mrbsetenv
317 46 Lynn Garren
mrb t -jN
318 46 Lynn Garren
</pre>
319 46 Lynn Garren
320 46 Lynn Garren
321 47 Lynn Garren
h3(count). Getting ready to the release
322 1 Lynn Garren
323 1 Lynn Garren
Make sure you update the nutools version.  
324 1 Lynn Garren
<pre>
325 47 Lynn Garren
mrb uv nutools <new tag>
326 47 Lynn Garren
mrb uv nulite <new tag>
327 1 Lynn Garren
</pre>
328 1 Lynn Garren
329 47 Lynn Garren
h3(count).  Final build and test
330 47 Lynn Garren
331 47 Lynn Garren
Make one more test build to make sure there are no problems.  This build will also generate the scripts needed to build the nulite distribution.
332 47 Lynn Garren
<pre>
333 47 Lynn Garren
cd $MRB_BUILDDIR
334 47 Lynn Garren
mrb z (if the directory is not empty)
335 47 Lynn Garren
mrbsetenv
336 47 Lynn Garren
mrb t -jN
337 47 Lynn Garren
</pre>
338 47 Lynn Garren
Copy the nulite distribution scripts to SciSoft. It is advisable to compare them to the scripts for the previous distribution before uploading.
339 47 Lynn Garren
<pre>
340 47 Lynn Garren
cd $MRB_BUILDDIR/nutools/bundle
341 47 Lynn Garren
copyToSciSoft nulite*
342 47 Lynn Garren
</pre>
343 47 Lynn Garren
344 47 Lynn Garren
h3(count). Tag 
345 47 Lynn Garren
346 47 Lynn Garren
Make sure you commit all changes in nutools.  Once that is done, generate the tag and update.
347 47 Lynn Garren
<pre>
348 47 Lynn Garren
cd $MRB_SOURCE/nutools
349 47 Lynn Garren
git commit -m"..." file(s)
350 47 Lynn Garren
git tag -a -m"..." <new tag>
351 47 Lynn Garren
git push origin lar_v2_27_br <new tag>
352 47 Lynn Garren
</pre>
353 10 Lynn Garren
354 37 Lynn Garren
h2(count). Build the nulite distribution
355 1 Lynn Garren
356 37 Lynn Garren
There are several Jenkins jobs that build the nulite distribution.
357 1 Lynn Garren
Choose "Build with Parameters" and edit NUVER appropriately.
358 42 Lynn Garren
359 42 Lynn Garren
* Use "build-nulite":https://buildmaster.fnal.gov/buildmaster/view/Art/job/build-nulite/ for the head nutools.
360 42 Lynn Garren
* Use "build-nulite-s85":https://buildmaster.fnal.gov/buildmaster/view/Art/job/build-nulite-s85/ for NOvA.
361 42 Lynn Garren
* Use "build-nulite-s78":https://buildmaster.fnal.gov/buildmaster/view/Art/job/build-nulite-s78/ to make a distribution for the MicroBooNE v08_05_00_xx larsoft patch releases.
362 10 Lynn Garren
363 37 Lynn Garren
h3(count).  When the build finishes
364 37 Lynn Garren
365 37 Lynn Garren
Sometimes a problem shows up the the build that was not caught during testing.  If this happens, make the fix and rebuild.  It is OK to move a tag for code in redmine if there is no build yet on SciSoft.  If there is already a build on SciSoft, a new tag is required.
366 37 Lynn Garren
367 38 Lynn Garren
When the job finishes successfully, the manifests and tarballs will be in "Last Successful Artifacts" for each matrix entry.  Click on the green ball to see them.  Since it is tedious at best to copy directly from Jenkins, use copyFromJenkins.  You can do this on any machine, but I prefer to use scisoftportal.  Make sure you download to an empty directory so you can use the wildcard.
368 43 Lynn Garren
369 43 Lynn Garren
The list of qualifiers will be defined by the Jenkins build job.  This example is for build-nulite with s94.
370 37 Lynn Garren
<pre>
371 37 Lynn Garren
ssh scisoftportal.fnal.gov
372 37 Lynn Garren
mkdir tmp
373 37 Lynn Garren
cd tmp
374 1 Lynn Garren
/nasroot/SciSoft/bundles/tools/copyFromJenkins -N -q s94-c7 -q s94-c7-py2 -q s94-e19 -q s94-e19-py2 build-nulite
375 38 Lynn Garren
/nasroot/SciSoft/bundles/tools/copyToSciSoft *
376 37 Lynn Garren
</pre>
377 39 Lynn Garren
378 39 Lynn Garren
h2(count). Notification
379 39 Lynn Garren
380 39 Lynn Garren
Notify the appropriate people that the build is complete and available on SciSoft.
381 39 Lynn Garren
* If the build was made in response to a redmine ticket, update the ticket.
382 39 Lynn Garren
* If the build is needed for an upcoming larsoft release, send email to scisoft-team@fnal.gov.