Project

General

Profile

Notes for NuTools release managers » History » Version 27

Lynn Garren, 02/04/2020 11:21 AM

1 1 Lynn Garren
h1. Notes for NuTools release managers
2 1 Lynn Garren
3 4 Lynn Garren
{{>TOC}}
4 4 Lynn Garren
5 6 Lynn Garren
h2(count). Updating one package
6 1 Lynn Garren
7 1 Lynn Garren
* If you only need a new version of a single package, such as nugen, then simply checkout that package, edit, tag and build.
8 1 Lynn Garren
9 5 Lynn Garren
* If you want a complete nulite distribution with the new package, then you would also checkout nutools and follow the instructions for updating the nu suite.
10 2 Lynn Garren
11 2 Lynn Garren
h2(count). The nu suite
12 2 Lynn Garren
13 2 Lynn Garren
The nu suite consists of nutools, nuevdb, nug4, nugen, nurandom, and nusimdata.
14 2 Lynn Garren
15 26 Lynn Garren
16 26 Lynn Garren
h3(count). Branch names
17 26 Lynn Garren
18 26 Lynn Garren
* Branches for NOvA are of the form nova_v3_01_br
19 26 Lynn Garren
* Branches for LArSoft are of the form lar_v2_27_br
20 26 Lynn Garren
* Because LArSoft moves ahead with art more frequently than NOvA, most larsoft distributions are made off the head of develop.
21 26 Lynn Garren
22 2 Lynn Garren
h2(count).  Tools
23 2 Lynn Garren
24 19 Lynn Garren
* mrb v3_04_04 or later 
25 19 Lynn Garren
** This version adds the nu_suite gitCheckout option and fixes some bugs in mrb uv.
26 2 Lynn Garren
* buildFW (used by the Jenkins build)
27 7 Lynn Garren
* copyFromJenkins
28 7 Lynn Garren
* copyToSciSoft
29 8 Lynn Garren
* pullProducts
30 25 Lynn Garren
* pullPackage
31 1 Lynn Garren
* larreltools v1_08_00 or later
32 25 Lynn Garren
** This package was created for LArSoft, but contains common tools.
33 1 Lynn Garren
** dogit
34 5 Lynn Garren
** startNuLiteRel
35 9 Lynn Garren
** updateVersion
36 1 Lynn Garren
37 27 Lynn Garren
h2(count). Updating the entire nutools suite from the head of develop
38 9 Lynn Garren
39 26 Lynn Garren
These instructions presume that the work is done on scisoftbuild01.fnal.gov.  These instructions also presume that the head of develop will be used. 
40 10 Lynn Garren
41 14 Lynn Garren
h3(count).  update "third party" products
42 14 Lynn Garren
43 15 Lynn Garren
Before building the nu suite, new releases of "third party" products need to be available.  Our definition of "third party products" is any product that does not use cetbuildtools or cetmodules.
44 15 Lynn Garren
45 15 Lynn Garren
This example updates the nu suite to use art 3.04.00.  python3 is the default. This means that the py3 qualifier is no longer used.  Instead, the py2 qualifier is used to specify a build with python v2_7_15a.  Because of this change, root has been updated to v6_18_04c.  Many products in the nulite distribution will need new releases with updated table files.  
46 15 Lynn Garren
47 21 Lynn Garren
* genie v3_00_06e
48 21 Lynn Garren
* dk2nudata v1_08_00f
49 21 Lynn Garren
* dk2nugenie v1_08_00h
50 14 Lynn Garren
51 10 Lynn Garren
h3(count). setup the working environment
52 10 Lynn Garren
53 9 Lynn Garren
<pre>
54 9 Lynn Garren
source /products/setup
55 9 Lynn Garren
setup mrb
56 9 Lynn Garren
setup larreltools
57 9 Lynn Garren
export PATH=$LARRELTOOLS_DIR/expert:${PATH}
58 9 Lynn Garren
export MRB_PROJECT=nu
59 9 Lynn Garren
</pre>
60 9 Lynn Garren
61 1 Lynn Garren
h3(count). make a working directory and checkout the code
62 1 Lynn Garren
63 26 Lynn Garren
Use the startNuLiteRel script.
64 26 Lynn Garren
<pre>
65 26 Lynn Garren
Usage: startNuLiteRel <working_dir> <new tag> <base qualifier>
66 26 Lynn Garren
   This script will do the following:
67 26 Lynn Garren
       mkdir <working_dir>/<tag>
68 26 Lynn Garren
       run newDev
69 26 Lynn Garren
       git clone nulite repositories from redmine
70 26 Lynn Garren
   The base qualifier should be one of e17, e19, c7 etc.
71 26 Lynn Garren
</pre>
72 26 Lynn Garren
For instance:
73 10 Lynn Garren
<pre>
74 25 Lynn Garren
startNuLiteRel `pwd` v3_06_00 e19
75 25 Lynn Garren
startNuLiteRel `pwd` v3_06_00 c7
76 13 Lynn Garren
</pre>
77 13 Lynn Garren
78 13 Lynn Garren
79 12 Lynn Garren
h3(count). update, build, and test the code
80 12 Lynn Garren
81 12 Lynn Garren
Create a temporary release branch.  This workflow mimics, but does not use, gitflow.  This allows us more flexibility when dealing with multiple repositories.
82 12 Lynn Garren
<pre>
83 12 Lynn Garren
source v3_06_00/e19p/localProducts_nu_v3_06_00_e19_prof/setup
84 12 Lynn Garren
cd $MRB_SOURCE
85 12 Lynn Garren
tagLAr start $MRB_PROJECT_VERSION
86 12 Lynn Garren
</pre>
87 12 Lynn Garren
Now, we update the code.  This example updates the head of the nutools suite to use art 3.04.
88 13 Lynn Garren
<pre>
89 13 Lynn Garren
mrb uv canvas_root_io v1_05_00
90 16 Lynn Garren
mrb uv art v3_04_00
91 13 Lynn Garren
mrb uv art_root_io v1_02_00
92 16 Lynn Garren
mrb uv gallery v1_14_00
93 17 Lynn Garren
mrb uv toyExperiment v0_05_00
94 1 Lynn Garren
mrb uv ifdh_art v2_10_01
95 22 Lynn Garren
</pre>
96 24 Lynn Garren
In this case, we use mrb cq (changeQualifier) from py3 to py2 and also update the version_set.
97 22 Lynn Garren
<pre>
98 22 Lynn Garren
mrb uv version_set s94
99 22 Lynn Garren
mrb cq py3 py2
100 13 Lynn Garren
</pre>
101 23 Lynn Garren
Make sure the required products are available.  We install the critic distribution in /products on scisoftbuild01.  If there are local builds of other products, make sure they are installed in any private product directory.  Add symbolic links to the product in $MRB_INSTALL.
102 23 Lynn Garren
<pre>
103 23 Lynn Garren
cd $MRB_INSTALL
104 23 Lynn Garren
ln -s /home/garren/scratch/products/genie
105 23 Lynn Garren
etc.
106 23 Lynn Garren
</pre>
107 23 Lynn Garren
Build and test.  Iterate as necessary until everything builds cleanly.
108 23 Lynn Garren
<pre>
109 23 Lynn Garren
cd $MRB_BUILDDIR
110 23 Lynn Garren
mrb z (when necessary)
111 23 Lynn Garren
mrbsetenv
112 23 Lynn Garren
mrb t -jN
113 23 Lynn Garren
</pre>
114 23 Lynn Garren
115 23 Lynn Garren
h3(count). Tagging the release
116 23 Lynn Garren
117 23 Lynn Garren
First update the version.  In this case, the art release and accompanying qualifiers have changed.  The convention is to update the minor version of the packages in the nu suite.
118 23 Lynn Garren
<pre>
119 23 Lynn Garren
updateVersion minor
120 23 Lynn Garren
</pre>
121 1 Lynn Garren
It's wise to have a look at each package and make sure the changes are what you expect.
122 27 Lynn Garren
123 27 Lynn Garren
124 27 Lynn Garren
h2(count). Updating part of the nutools suite.
125 27 Lynn Garren
126 27 Lynn Garren
127 27 Lynn Garren
h3(count).  update "third party" products
128 27 Lynn Garren
129 27 Lynn Garren
Before building the nu suite, new releases of "third party" products need to be available.  Our definition of "third party products" is any product that does not use cetbuildtools or cetmodules.
130 27 Lynn Garren
131 27 Lynn Garren
This example updates the nu suite to use ifdhc v2_5_4, libwda v2_27_0, and ifdh_art v2_10_02.  
132 27 Lynn Garren
133 27 Lynn Garren
python3 is the default. This means that the py3 qualifier is no longer used.  Instead, the py2 qualifier is used to specify a build with python v2_7_15a.
134 27 Lynn Garren
135 27 Lynn Garren
h3(count). setup the working environment
136 27 Lynn Garren
137 27 Lynn Garren
<pre>
138 27 Lynn Garren
source /products/setup
139 27 Lynn Garren
setup mrb
140 27 Lynn Garren
setup larreltools
141 27 Lynn Garren
export PATH=$LARRELTOOLS_DIR/expert:${PATH}
142 27 Lynn Garren
export MRB_PROJECT=nu
143 27 Lynn Garren
</pre>
144 27 Lynn Garren
145 27 Lynn Garren
h3(count). make a working directory and checkout the code
146 27 Lynn Garren
147 27 Lynn Garren
Use the startNuLiteRel script.
148 27 Lynn Garren
<pre>
149 27 Lynn Garren
Usage: startNuLiteRel <working_dir> <new tag> <base qualifier>
150 27 Lynn Garren
   This script will do the following:
151 27 Lynn Garren
       mkdir <working_dir>/<tag>
152 27 Lynn Garren
       run newDev
153 27 Lynn Garren
       git clone nulite repositories from redmine
154 27 Lynn Garren
   The base qualifier should be one of e17, e19, c7 etc.
155 27 Lynn Garren
</pre>
156 27 Lynn Garren
For instance:
157 27 Lynn Garren
<pre>
158 27 Lynn Garren
startNuLiteRel `pwd` v3_06_01 e19
159 27 Lynn Garren
startNuLiteRel `pwd` v3_06_01 c7
160 27 Lynn Garren
</pre>
161 27 Lynn Garren
162 27 Lynn Garren
163 27 Lynn Garren
h3(count). update, build, and test the code
164 23 Lynn Garren
165 12 Lynn Garren
166 1 Lynn Garren
h2(count). Updating the nutools suite for NOvA
167 12 Lynn Garren
168 12 Lynn Garren
All NOvA releases are tagged and built off a special branch.
169 9 Lynn Garren
170 11 Lynn Garren
h3(count). setup the working environment
171 9 Lynn Garren
172 9 Lynn Garren
<pre>
173 9 Lynn Garren
source /cvmfs/nova.opensciencegrid.org/externals/setup
174 9 Lynn Garren
export PRODUCTS=$PRODUCTS:/cvmfs/larsoft.opensciencegrid.org/products
175 9 Lynn Garren
setup mrb
176 9 Lynn Garren
setup larreltools
177 9 Lynn Garren
export PATH=$LARRELTOOLS_DIR/expert:${PATH}
178 9 Lynn Garren
export MRB_PROJECT=nu
179 9 Lynn Garren
</pre>
180 10 Lynn Garren
181 10 Lynn Garren
h3(count). make a working directory and checkout the code
182 10 Lynn Garren
183 10 Lynn Garren
<pre>
184 10 Lynn Garren
makeNuLiteWorkDir `pwd` v3_01_04 e17
185 10 Lynn Garren
makeNuLiteWorkDir `pwd` v3_01_04 c2
186 10 Lynn Garren
</pre>
187 10 Lynn Garren
188 9 Lynn Garren
189 9 Lynn Garren
190 9 Lynn Garren
191 5 Lynn Garren
h2(count). just saving for reference
192 5 Lynn Garren
193 5 Lynn Garren
* If you want a complete nulite distribution with the new package, then you would also checkout nutools and follow the instructions for updating the nu suite.
194 5 Lynn Garren
** Edit the package version and the nudist version in nutools/bundle/CMakeLists.txt.
195 5 Lynn Garren
** The nudist version must be the same as the nutools version, but with an alphabet character added to the end.
196 5 Lynn Garren
*** For instance, v3_01_03 would become v3_01_03a, v3_01_03b would become v3_01_03c, etc.
197 5 Lynn Garren
** Build nutools and use the generated nutools/bundle/nu* files. (See below for more information.)