Project

General

Profile

How to tag and build a LArSoft patch release » History » Version 33

Herbert Greenlee, 03/04/2020 10:07 AM

1 1 Lynn Garren
h1. How to tag and build a LArSoft patch release
2 1 Lynn Garren
3 3 Lynn Garren
{{>TOC}}
4 3 Lynn Garren
5 10 Lynn Garren
[[Policy for development from a tagged release]]
6 1 Lynn Garren
Experiments will build patch releases themselves with help from the LArSoft release manager.
7 2 Lynn Garren
There will be a very limited set of people with privileges to do this.  
8 7 Lynn Garren
We ask that the experiment release managers to whom these privileges have been granted do not grant privileges to anyone else.  Instead request permissions from LArSoft.
9 6 Lynn Garren
Permissions will be removed when someone ceases to have this responsibility.
10 2 Lynn Garren
11 10 Lynn Garren
* Permissions include:
12 10 Lynn Garren
** larsoft redmine manager role
13 10 Lynn Garren
** ability to login to larsoft cvmfs
14 10 Lynn Garren
** ability to upload to SciSoft with the larsoft role
15 12 Lynn Garren
** developer status for larreltools
16 22 Lynn Garren
** abiltiy to run Jenkins jobs
17 10 Lynn Garren
18 32 Lynn Garren
Please make a service desk ticket asking for these permissions.  Use the SciSoft at Service Now link at the bottom of the "SciSoft home page":https://scisoft.fnal.gov/.
19 32 Lynn Garren
20 13 Lynn Garren
[[LArSoft responsiblities for patch releases]]
21 13 Lynn Garren
22 11 Lynn Garren
h2(count). Request a release
23 11 Lynn Garren
24 11 Lynn Garren
* Before starting any patch release, fill out a redmine issue requesting the release with all pertinent information.
25 10 Lynn Garren
26 10 Lynn Garren
27 2 Lynn Garren
h2(count). Tools
28 2 Lynn Garren
29 2 Lynn Garren
* Scripts used to build the patch release will be found in the larreltools product.  
30 19 Lynn Garren
* *larreltools* is not part of the standard distribution, but will be available from SciSoft and in cvmfs.
31 16 Lynn Garren
** Always use the latest release, which will be declared current.
32 20 Lynn Garren
** More [[larreltools:|information]]
33 4 Lynn Garren
34 2 Lynn Garren
h2(count). Initial setup
35 1 Lynn Garren
36 15 Lynn Garren
*This step is done once for any patch release set and will be done by the LArSoft release manager.*
37 9 Lynn Garren
38 4 Lynn Garren
h2(count). Making a patch release
39 2 Lynn Garren
40 2 Lynn Garren
* These notes presume that you do not have an existing directory
41 2 Lynn Garren
42 24 Lynn Garren
h3(count). Setup the working area
43 2 Lynn Garren
44 1 Lynn Garren
* setup the latest release of larreltools
45 33 Herbert Greenlee
* startPatchRel [-e <experiment] <working_dir> <new patch tag> <larsoft branch> [<experiment branch>]
46 24 Lynn Garren
** Recognized experiment names are argoneut, dune, lariat, icarus, sbnd, and uboone.
47 2 Lynn Garren
** This script will create a directory named "tag" under the "working_dir".
48 24 Lynn Garren
** This script will clone the larsoft and larsoftobj suites using the named branch.
49 24 Lynn Garren
** The same branch name will be used when cloning the experiment code unless a second branch name is supplied.
50 2 Lynn Garren
** The directory structure will include a single source code directory with subdirectories for e17 and c2 debug and prof builds.
51 24 Lynn Garren
** Use "dogit branch" to double check that you are on the expected branch of all packages.
52 2 Lynn Garren
53 4 Lynn Garren
h3(count). Local development
54 2 Lynn Garren
55 2 Lynn Garren
* Before integrating any feature branches, make sure that the release builds as is and that the unit tests work.
56 2 Lynn Garren
** Make sure to build and test both e17 and c2.   Usually the prof build is sufficient for testing.  prof is preferred over debug, which sometimes masks problems.
57 2 Lynn Garren
<pre>
58 2 Lynn Garren
source local*/setup
59 2 Lynn Garren
cd $MRB_BUILDDIR
60 2 Lynn Garren
mrb t -jN
61 2 Lynn Garren
</pre>
62 2 Lynn Garren
* Next make changes and integrate feature branches
63 2 Lynn Garren
** Because feature branches may conflict or not have been thoroughly tested, we recommend merging one feature branch set at a time.  Build and test after each set.
64 2 Lynn Garren
65 4 Lynn Garren
h3(count). Update the package versions
66 2 Lynn Garren
67 2 Lynn Garren
* Do this once all changes have been made.
68 25 Lynn Garren
69 25 Lynn Garren
* Use the *updatePatchVersion* script from larreltools.
70 25 Lynn Garren
** This script takes no arguments.
71 25 Lynn Garren
** This script will call "mrb uv" to update each patch release by one.
72 25 Lynn Garren
** If there was no previous patch release, the version will be vaa_bb_cc_01
73 25 Lynn Garren
74 2 Lynn Garren
* ONLY update the version if there is a change.
75 2 Lynn Garren
* cd to each directory separately
76 2 Lynn Garren
* git diff LARSOFT_SUITE_vaa_bb_cc[_dd]
77 2 Lynn Garren
** You are comparing against the previous patch release in this set.
78 1 Lynn Garren
** If this is the first patch release in the set, compare against the original release.
79 25 Lynn Garren
** Sometimes the only change is an updated dependency.
80 25 Lynn Garren
** If there are no changes, use mrb uv to restore the previous release version and then move this package out of the way.
81 1 Lynn Garren
<pre>
82 1 Lynn Garren
cd $MRB_SOURCE
83 1 Lynn Garren
mkdir ../notag
84 16 Lynn Garren
mv xxx ../notag
85 17 Lynn Garren
mrb uc
86 16 Lynn Garren
</pre>
87 16 Lynn Garren
** Move all experiment code out of the way.  (_mv ub* ../notag/_ for MicroBooNE)
88 1 Lynn Garren
89 26 Lynn Garren
* As of mrb v3_02_01, the "mrb uv" command will update larsoftobj/bundle/CMakeLists.txt, larsoft/releaseDB/CMakeLists, and <my experiment>/releaseDB/CMakeLists.txt as well as ups/product_deps.
90 30 Lynn Garren
** commit changes in the releaseDB or bundle subdirectories, but do not commit changes in ups/product_deps.
91 25 Lynn Garren
* DO NOT commit the changes to ups/product_deps.  tagPatchRel will commit ups/product_deps.
92 2 Lynn Garren
93 27 Lynn Garren
h3(count).  Dependency order
94 2 Lynn Garren
95 27 Lynn Garren
* Within the larsoftobj suite
96 27 Lynn Garren
** larcoreobj
97 27 Lynn Garren
** larcorealg
98 27 Lynn Garren
** lardataobj
99 27 Lynn Garren
** lardataalg
100 27 Lynn Garren
** larsoftobj
101 27 Lynn Garren
102 27 Lynn Garren
* All larsoft packages
103 27 Lynn Garren
** larcoreobj
104 27 Lynn Garren
** larcorealg
105 27 Lynn Garren
** larcore
106 27 Lynn Garren
** lardataobj
107 27 Lynn Garren
** lardataalg
108 27 Lynn Garren
** larsoftobj
109 27 Lynn Garren
** lardata
110 27 Lynn Garren
** larevt
111 31 Lynn Garren
** larwirecell (depends on larevt)
112 31 Lynn Garren
** larg4 (depends on larevt)
113 31 Lynn Garren
** larsim (depends on larevt)
114 27 Lynn Garren
** larreco
115 31 Lynn Garren
** larana (depends on larreco)
116 31 Lynn Garren
** larpandora (depends on larreco)
117 31 Lynn Garren
** lareventdisplay (depends on larreco)
118 31 Lynn Garren
** larexamples (depends on larreco)
119 27 Lynn Garren
** larsoft
120 27 Lynn Garren
121 27 Lynn Garren
122 27 Lynn Garren
123 27 Lynn Garren
124 27 Lynn Garren
125 4 Lynn Garren
h3(count).  Tagging the release
126 2 Lynn Garren
127 2 Lynn Garren
* Do a final build and test with just the packages you expect to tag.  
128 25 Lynn Garren
** This step will also create the necessary build config files.
129 2 Lynn Garren
** If there are changes in both larsoft and larsoftobj, double check for consistency.
130 2 Lynn Garren
<pre>
131 1 Lynn Garren
cd $MRB_BUILDDIR/larsoft
132 2 Lynn Garren
diff releaseDB ../larsoftobj/bundle | grep diff
133 2 Lynn Garren
diff releaseDB ../larsoftobj/bundle | grep Only
134 2 Lynn Garren
</pre>
135 2 Lynn Garren
** If all is good, copy the cfg scripts to SciSoft
136 2 Lynn Garren
<pre>
137 2 Lynn Garren
cd $MRB_BUILDDIR/larsoft/releaseDB
138 2 Lynn Garren
copyToSciSoft lar*
139 25 Lynn Garren
</pre>
140 2 Lynn Garren
* Now tag
141 2 Lynn Garren
** tagPatchRel <existing branch> <new larsoft tag>
142 2 Lynn Garren
** tagPatchRel will commit ups/product_deps with an appropriate message, make the tag, and push both tag and branch to origin.
143 4 Lynn Garren
** Use "dogit status" to doublecheck.
144 2 Lynn Garren
** *IMPORTANT*:  patch release tags are never merged with the master branch
145 2 Lynn Garren
146 2 Lynn Garren
h3(count).  Official build
147 2 Lynn Garren
148 21 Lynn Garren
* build on Jenkins
149 21 Lynn Garren
* Use special patch release build jobs.
150 1 Lynn Garren
** This avoids possible conflict with normal release builds.
151 1 Lynn Garren
** "larpatch-slf":https://buildmaster.fnal.gov/buildmaster/view/LArSoft/job/larpatch-slf/
152 1 Lynn Garren
** "larpatch-mac":https://buildmaster.fnal.gov/buildmaster/view/LArSoft/job/larpatch-mac/
153 5 Lynn Garren
* use copyFromJenkins to download the tarballs and release manifests from Jenkins
154 5 Lynn Garren
* use copyToSciSoft to upload the tarballs and release manifests to SciSoft
155 5 Lynn Garren
** This step requires extra privileges.
156 2 Lynn Garren
157 5 Lynn Garren
h3(count).  Release Notes and Cross Package Tag
158 2 Lynn Garren
159 2 Lynn Garren
* Once the build is complete for ALL variants (SLF and macOS), make the cross package tag.
160 18 Lynn Garren
** We wait because sometimes a problem is not found until building and we need to move a tag.
161 2 Lynn Garren
** This step requires redmine manager persmission for larsoft
162 18 Lynn Garren
<pre>
163 2 Lynn Garren
cp-patch-tag <larsoft tag> <larsoftobj tag>
164 23 Lynn Garren
</pre>
165 2 Lynn Garren
*** It would be nice to fix cp-patch-tag so it deduces the larsoftobj tag
166 2 Lynn Garren
* Make the release notes
167 1 Lynn Garren
** makePatchRelNotes <working_directory> <tag> <previous tag>
168 27 Lynn Garren
*** This script will create a subdirectory named tag under the working_directory and generate a text file.
169 1 Lynn Garren
** cd into the newly created tag directory
170 1 Lynn Garren
** cat ReleaseNotes-vaa_bb_cc_dd
171 27 Lynn Garren
*** The first line of this output is a table file entry.  Cut and paste into the top of the [[LArSoft_release_list|release list]] 
172 2 Lynn Garren
*** The remainder of the output is used to create the release note itself.
173 27 Lynn Garren
**** Cut and past everything from "h1. LArSoft ..." to the bottom of the file into the empty release note entry.
174 27 Lynn Garren
**** Edit the information in part as appropriate.
175 1 Lynn Garren
**** Do not edit anything under the "Change List" section.
176 1 Lynn Garren
**** It is important to use cat so that there are no unexpected line breaks.
177 5 Lynn Garren
178 5 Lynn Garren
h3(count). Upload to cvmfs
179 5 Lynn Garren
180 5 Lynn Garren
* This step requires login privileges.
181 5 Lynn Garren
* ssh cvmfslarsoft@oasiscfs.fnal.gov
182 5 Lynn Garren
* The README provides an overview of the necessary steps.
183 5 Lynn Garren
** start a transaction
184 5 Lynn Garren
** install using installBundle.sh
185 5 Lynn Garren
** publish
186 5 Lynn Garren
* installBundle.sh larsoft vaa_bb_cc_zz sNN-eY
187 1 Lynn Garren
** Note that if you call installBundle.sh with sNN-e17, it will install both e17 and c2 variants.
188 5 Lynn Garren
** The script will attempt to install the release for all supported platforms and clean up after itself.
189 5 Lynn Garren
** *If some expected distributions are not available, then manifests will be left in the .working subdirectory. Check and remove them if necessary.*
190 27 Lynn Garren
* installBundleSLF.sh is the same as installBundle.sh, but only installs for SLF6 and SLF7.
191 28 Lynn Garren
* Do not leave a transaction open.  You can abort the changes you made if there is a problem.
192 29 Lynn Garren
193 31 Lynn Garren
h2(count). Presentations
194 29 Lynn Garren
195 29 Lynn Garren
A patch release tutorial was given at the "2019 LArSoft workshop":https://indico.fnal.gov/event/20453/session/7/material/1/0.pdf