How to tag and build a LArSoft patch release » History » Version 32
Lynn Garren, 11/05/2019 10:08 AM
1 | 1 | Lynn Garren | h1. How to tag and build a LArSoft patch release |
---|---|---|---|
2 | 1 | Lynn Garren | |
3 | 3 | Lynn Garren | {{>TOC}} |
4 | 3 | Lynn Garren | |
5 | 10 | Lynn Garren | [[Policy for development from a tagged release]] |
6 | 1 | Lynn Garren | Experiments will build patch releases themselves with help from the LArSoft release manager. |
7 | 2 | Lynn Garren | There will be a very limited set of people with privileges to do this. |
8 | 7 | Lynn Garren | We ask that the experiment release managers to whom these privileges have been granted do not grant privileges to anyone else. Instead request permissions from LArSoft. |
9 | 6 | Lynn Garren | Permissions will be removed when someone ceases to have this responsibility. |
10 | 2 | Lynn Garren | |
11 | 10 | Lynn Garren | * Permissions include: |
12 | 10 | Lynn Garren | ** larsoft redmine manager role |
13 | 10 | Lynn Garren | ** ability to login to larsoft cvmfs |
14 | 10 | Lynn Garren | ** ability to upload to SciSoft with the larsoft role |
15 | 12 | Lynn Garren | ** developer status for larreltools |
16 | 22 | Lynn Garren | ** abiltiy to run Jenkins jobs |
17 | 10 | Lynn Garren | |
18 | 32 | Lynn Garren | Please make a service desk ticket asking for these permissions. Use the SciSoft at Service Now link at the bottom of the "SciSoft home page":https://scisoft.fnal.gov/. |
19 | 32 | Lynn Garren | |
20 | 13 | Lynn Garren | [[LArSoft responsiblities for patch releases]] |
21 | 13 | Lynn Garren | |
22 | 11 | Lynn Garren | h2(count). Request a release |
23 | 11 | Lynn Garren | |
24 | 11 | Lynn Garren | * Before starting any patch release, fill out a redmine issue requesting the release with all pertinent information. |
25 | 10 | Lynn Garren | |
26 | 10 | Lynn Garren | |
27 | 2 | Lynn Garren | h2(count). Tools |
28 | 2 | Lynn Garren | |
29 | 2 | Lynn Garren | * Scripts used to build the patch release will be found in the larreltools product. |
30 | 19 | Lynn Garren | * *larreltools* is not part of the standard distribution, but will be available from SciSoft and in cvmfs. |
31 | 16 | Lynn Garren | ** Always use the latest release, which will be declared current. |
32 | 20 | Lynn Garren | ** More [[larreltools:|information]] |
33 | 4 | Lynn Garren | |
34 | 2 | Lynn Garren | h2(count). Initial setup |
35 | 1 | Lynn Garren | |
36 | 15 | Lynn Garren | *This step is done once for any patch release set and will be done by the LArSoft release manager.* |
37 | 9 | Lynn Garren | |
38 | 4 | Lynn Garren | h2(count). Making a patch release |
39 | 2 | Lynn Garren | |
40 | 2 | Lynn Garren | * These notes presume that you do not have an existing directory |
41 | 2 | Lynn Garren | |
42 | 24 | Lynn Garren | h3(count). Setup the working area |
43 | 2 | Lynn Garren | |
44 | 1 | Lynn Garren | * setup the latest release of larreltools |
45 | 24 | Lynn Garren | * startPatchRel <working_dir> <experiment> <new patch tag> <larsoft branch> [<experiment branch>] |
46 | 24 | Lynn Garren | ** Recognized experiment names are argoneut, dune, lariat, icarus, sbnd, and uboone. |
47 | 2 | Lynn Garren | ** This script will create a directory named "tag" under the "working_dir". |
48 | 24 | Lynn Garren | ** This script will clone the larsoft and larsoftobj suites using the named branch. |
49 | 24 | Lynn Garren | ** The same branch name will be used when cloning the experiment code unless a second branch name is supplied. |
50 | 2 | Lynn Garren | ** The directory structure will include a single source code directory with subdirectories for e17 and c2 debug and prof builds. |
51 | 24 | Lynn Garren | ** Use "dogit branch" to double check that you are on the expected branch of all packages. |
52 | 2 | Lynn Garren | |
53 | 4 | Lynn Garren | h3(count). Local development |
54 | 2 | Lynn Garren | |
55 | 2 | Lynn Garren | * Before integrating any feature branches, make sure that the release builds as is and that the unit tests work. |
56 | 2 | Lynn Garren | ** Make sure to build and test both e17 and c2. Usually the prof build is sufficient for testing. prof is preferred over debug, which sometimes masks problems. |
57 | 2 | Lynn Garren | <pre> |
58 | 2 | Lynn Garren | source local*/setup |
59 | 2 | Lynn Garren | cd $MRB_BUILDDIR |
60 | 2 | Lynn Garren | mrb t -jN |
61 | 2 | Lynn Garren | </pre> |
62 | 2 | Lynn Garren | * Next make changes and integrate feature branches |
63 | 2 | Lynn Garren | ** Because feature branches may conflict or not have been thoroughly tested, we recommend merging one feature branch set at a time. Build and test after each set. |
64 | 2 | Lynn Garren | |
65 | 4 | Lynn Garren | h3(count). Update the package versions |
66 | 2 | Lynn Garren | |
67 | 2 | Lynn Garren | * Do this once all changes have been made. |
68 | 25 | Lynn Garren | |
69 | 25 | Lynn Garren | * Use the *updatePatchVersion* script from larreltools. |
70 | 25 | Lynn Garren | ** This script takes no arguments. |
71 | 25 | Lynn Garren | ** This script will call "mrb uv" to update each patch release by one. |
72 | 25 | Lynn Garren | ** If there was no previous patch release, the version will be vaa_bb_cc_01 |
73 | 25 | Lynn Garren | |
74 | 2 | Lynn Garren | * ONLY update the version if there is a change. |
75 | 2 | Lynn Garren | * cd to each directory separately |
76 | 2 | Lynn Garren | * git diff LARSOFT_SUITE_vaa_bb_cc[_dd] |
77 | 2 | Lynn Garren | ** You are comparing against the previous patch release in this set. |
78 | 1 | Lynn Garren | ** If this is the first patch release in the set, compare against the original release. |
79 | 25 | Lynn Garren | ** Sometimes the only change is an updated dependency. |
80 | 25 | Lynn Garren | ** If there are no changes, use mrb uv to restore the previous release version and then move this package out of the way. |
81 | 1 | Lynn Garren | <pre> |
82 | 1 | Lynn Garren | cd $MRB_SOURCE |
83 | 1 | Lynn Garren | mkdir ../notag |
84 | 16 | Lynn Garren | mv xxx ../notag |
85 | 17 | Lynn Garren | mrb uc |
86 | 16 | Lynn Garren | </pre> |
87 | 16 | Lynn Garren | ** Move all experiment code out of the way. (_mv ub* ../notag/_ for MicroBooNE) |
88 | 1 | Lynn Garren | |
89 | 26 | Lynn Garren | * As of mrb v3_02_01, the "mrb uv" command will update larsoftobj/bundle/CMakeLists.txt, larsoft/releaseDB/CMakeLists, and <my experiment>/releaseDB/CMakeLists.txt as well as ups/product_deps. |
90 | 30 | Lynn Garren | ** commit changes in the releaseDB or bundle subdirectories, but do not commit changes in ups/product_deps. |
91 | 25 | Lynn Garren | * DO NOT commit the changes to ups/product_deps. tagPatchRel will commit ups/product_deps. |
92 | 2 | Lynn Garren | |
93 | 27 | Lynn Garren | h3(count). Dependency order |
94 | 2 | Lynn Garren | |
95 | 27 | Lynn Garren | * Within the larsoftobj suite |
96 | 27 | Lynn Garren | ** larcoreobj |
97 | 27 | Lynn Garren | ** larcorealg |
98 | 27 | Lynn Garren | ** lardataobj |
99 | 27 | Lynn Garren | ** lardataalg |
100 | 27 | Lynn Garren | ** larsoftobj |
101 | 27 | Lynn Garren | |
102 | 27 | Lynn Garren | * All larsoft packages |
103 | 27 | Lynn Garren | ** larcoreobj |
104 | 27 | Lynn Garren | ** larcorealg |
105 | 27 | Lynn Garren | ** larcore |
106 | 27 | Lynn Garren | ** lardataobj |
107 | 27 | Lynn Garren | ** lardataalg |
108 | 27 | Lynn Garren | ** larsoftobj |
109 | 27 | Lynn Garren | ** lardata |
110 | 27 | Lynn Garren | ** larevt |
111 | 31 | Lynn Garren | ** larwirecell (depends on larevt) |
112 | 31 | Lynn Garren | ** larg4 (depends on larevt) |
113 | 31 | Lynn Garren | ** larsim (depends on larevt) |
114 | 27 | Lynn Garren | ** larreco |
115 | 31 | Lynn Garren | ** larana (depends on larreco) |
116 | 31 | Lynn Garren | ** larpandora (depends on larreco) |
117 | 31 | Lynn Garren | ** lareventdisplay (depends on larreco) |
118 | 31 | Lynn Garren | ** larexamples (depends on larreco) |
119 | 27 | Lynn Garren | ** larsoft |
120 | 27 | Lynn Garren | |
121 | 27 | Lynn Garren | |
122 | 27 | Lynn Garren | |
123 | 27 | Lynn Garren | |
124 | 27 | Lynn Garren | |
125 | 4 | Lynn Garren | h3(count). Tagging the release |
126 | 2 | Lynn Garren | |
127 | 2 | Lynn Garren | * Do a final build and test with just the packages you expect to tag. |
128 | 25 | Lynn Garren | ** This step will also create the necessary build config files. |
129 | 2 | Lynn Garren | ** If there are changes in both larsoft and larsoftobj, double check for consistency. |
130 | 2 | Lynn Garren | <pre> |
131 | 1 | Lynn Garren | cd $MRB_BUILDDIR/larsoft |
132 | 2 | Lynn Garren | diff releaseDB ../larsoftobj/bundle | grep diff |
133 | 2 | Lynn Garren | diff releaseDB ../larsoftobj/bundle | grep Only |
134 | 2 | Lynn Garren | </pre> |
135 | 2 | Lynn Garren | ** If all is good, copy the cfg scripts to SciSoft |
136 | 2 | Lynn Garren | <pre> |
137 | 2 | Lynn Garren | cd $MRB_BUILDDIR/larsoft/releaseDB |
138 | 2 | Lynn Garren | copyToSciSoft lar* |
139 | 25 | Lynn Garren | </pre> |
140 | 2 | Lynn Garren | * Now tag |
141 | 2 | Lynn Garren | ** tagPatchRel <existing branch> <new larsoft tag> |
142 | 2 | Lynn Garren | ** tagPatchRel will commit ups/product_deps with an appropriate message, make the tag, and push both tag and branch to origin. |
143 | 4 | Lynn Garren | ** Use "dogit status" to doublecheck. |
144 | 2 | Lynn Garren | ** *IMPORTANT*: patch release tags are never merged with the master branch |
145 | 2 | Lynn Garren | |
146 | 2 | Lynn Garren | h3(count). Official build |
147 | 2 | Lynn Garren | |
148 | 21 | Lynn Garren | * build on Jenkins |
149 | 21 | Lynn Garren | * Use special patch release build jobs. |
150 | 1 | Lynn Garren | ** This avoids possible conflict with normal release builds. |
151 | 1 | Lynn Garren | ** "larpatch-slf":https://buildmaster.fnal.gov/buildmaster/view/LArSoft/job/larpatch-slf/ |
152 | 1 | Lynn Garren | ** "larpatch-mac":https://buildmaster.fnal.gov/buildmaster/view/LArSoft/job/larpatch-mac/ |
153 | 5 | Lynn Garren | * use copyFromJenkins to download the tarballs and release manifests from Jenkins |
154 | 5 | Lynn Garren | * use copyToSciSoft to upload the tarballs and release manifests to SciSoft |
155 | 5 | Lynn Garren | ** This step requires extra privileges. |
156 | 2 | Lynn Garren | |
157 | 5 | Lynn Garren | h3(count). Release Notes and Cross Package Tag |
158 | 2 | Lynn Garren | |
159 | 2 | Lynn Garren | * Once the build is complete for ALL variants (SLF and macOS), make the cross package tag. |
160 | 18 | Lynn Garren | ** We wait because sometimes a problem is not found until building and we need to move a tag. |
161 | 2 | Lynn Garren | ** This step requires redmine manager persmission for larsoft |
162 | 18 | Lynn Garren | <pre> |
163 | 2 | Lynn Garren | cp-patch-tag <larsoft tag> <larsoftobj tag> |
164 | 23 | Lynn Garren | </pre> |
165 | 2 | Lynn Garren | *** It would be nice to fix cp-patch-tag so it deduces the larsoftobj tag |
166 | 2 | Lynn Garren | * Make the release notes |
167 | 1 | Lynn Garren | ** makePatchRelNotes <working_directory> <tag> <previous tag> |
168 | 27 | Lynn Garren | *** This script will create a subdirectory named tag under the working_directory and generate a text file. |
169 | 1 | Lynn Garren | ** cd into the newly created tag directory |
170 | 1 | Lynn Garren | ** cat ReleaseNotes-vaa_bb_cc_dd |
171 | 27 | Lynn Garren | *** The first line of this output is a table file entry. Cut and paste into the top of the [[LArSoft_release_list|release list]] |
172 | 2 | Lynn Garren | *** The remainder of the output is used to create the release note itself. |
173 | 27 | Lynn Garren | **** Cut and past everything from "h1. LArSoft ..." to the bottom of the file into the empty release note entry. |
174 | 27 | Lynn Garren | **** Edit the information in part as appropriate. |
175 | 1 | Lynn Garren | **** Do not edit anything under the "Change List" section. |
176 | 1 | Lynn Garren | **** It is important to use cat so that there are no unexpected line breaks. |
177 | 5 | Lynn Garren | |
178 | 5 | Lynn Garren | h3(count). Upload to cvmfs |
179 | 5 | Lynn Garren | |
180 | 5 | Lynn Garren | * This step requires login privileges. |
181 | 5 | Lynn Garren | * ssh cvmfslarsoft@oasiscfs.fnal.gov |
182 | 5 | Lynn Garren | * The README provides an overview of the necessary steps. |
183 | 5 | Lynn Garren | ** start a transaction |
184 | 5 | Lynn Garren | ** install using installBundle.sh |
185 | 5 | Lynn Garren | ** publish |
186 | 5 | Lynn Garren | * installBundle.sh larsoft vaa_bb_cc_zz sNN-eY |
187 | 1 | Lynn Garren | ** Note that if you call installBundle.sh with sNN-e17, it will install both e17 and c2 variants. |
188 | 5 | Lynn Garren | ** The script will attempt to install the release for all supported platforms and clean up after itself. |
189 | 5 | Lynn Garren | ** *If some expected distributions are not available, then manifests will be left in the .working subdirectory. Check and remove them if necessary.* |
190 | 27 | Lynn Garren | * installBundleSLF.sh is the same as installBundle.sh, but only installs for SLF6 and SLF7. |
191 | 28 | Lynn Garren | * Do not leave a transaction open. You can abort the changes you made if there is a problem. |
192 | 29 | Lynn Garren | |
193 | 31 | Lynn Garren | h2(count). Presentations |
194 | 29 | Lynn Garren | |
195 | 29 | Lynn Garren | A patch release tutorial was given at the "2019 LArSoft workshop":https://indico.fnal.gov/event/20453/session/7/material/1/0.pdf |