Project

General

Profile

How to tag and build a LArSoft patch release » History » Version 11

Lynn Garren, 02/27/2019 02:11 PM

1 1 Lynn Garren
h1. How to tag and build a LArSoft patch release
2 1 Lynn Garren
3 3 Lynn Garren
{{>TOC}}
4 3 Lynn Garren
5 10 Lynn Garren
[[Policy for development from a tagged release]]
6 1 Lynn Garren
Experiments will build patch releases themselves with help from the LArSoft release manager.
7 2 Lynn Garren
There will be a very limited set of people with privileges to do this.  
8 7 Lynn Garren
We ask that the experiment release managers to whom these privileges have been granted do not grant privileges to anyone else.  Instead request permissions from LArSoft.
9 6 Lynn Garren
Permissions will be removed when someone ceases to have this responsibility.
10 2 Lynn Garren
11 10 Lynn Garren
* Permissions include:
12 10 Lynn Garren
** larsoft redmine manager role
13 10 Lynn Garren
** ability to login to larsoft cvmfs
14 10 Lynn Garren
** ability to upload to SciSoft with the larsoft role
15 10 Lynn Garren
16 11 Lynn Garren
h2(count). Request a release
17 11 Lynn Garren
18 11 Lynn Garren
* Before starting any patch release, fill out a redmine issue requesting the release with all pertinent information.
19 10 Lynn Garren
20 10 Lynn Garren
21 2 Lynn Garren
h2(count). Tools
22 2 Lynn Garren
23 2 Lynn Garren
* Scripts used to build the patch release will be found in the larreltools product.  
24 2 Lynn Garren
* larreltools is not part of the standard distribution, but will be available from SciSoft and in cvmfs.
25 4 Lynn Garren
26 2 Lynn Garren
h2(count). Initial setup
27 1 Lynn Garren
28 10 Lynn Garren
*This step is done once for any patch release step and will be done by the LArSoft release manager.*
29 9 Lynn Garren
30 2 Lynn Garren
h3(count). Make branches for the release
31 2 Lynn Garren
32 2 Lynn Garren
* If a release is named vaa_bb_cc, branches will be created for the larsoft/larsoftobj suite with the name vaa_bb_cc_br.
33 2 Lynn Garren
<pre>
34 2 Lynn Garren
(setup the latest larreltools)
35 2 Lynn Garren
setup mrb
36 2 Lynn Garren
export MRB_PROJECT=larsoft
37 2 Lynn Garren
mrb newDev -v  vaa_bb_cc -q eN:prof 
38 2 Lynn Garren
source local*/setup
39 2 Lynn Garren
cd $MRB_SOURCE
40 2 Lynn Garren
mrb g -t LARSOFT_SUITE_vAA_BB_CC larsoft_suite
41 2 Lynn Garren
mrb g -t LARSOFT_SUITE_vAA_BB_CC larsoftobj_suite
42 2 Lynn Garren
dogit branch (check that you are on the head of the tag)
43 2 Lynn Garren
dogit "co -b vaa_bb_cc_br"
44 2 Lynn Garren
dogit "push -u origin vaa_bb_cc_br"
45 1 Lynn Garren
</pre>
46 8 Lynn Garren
** Note that some older patch releases may have used a slightly different branch naming scheme.
47 2 Lynn Garren
48 4 Lynn Garren
h3(count). Housecleaning
49 2 Lynn Garren
50 2 Lynn Garren
* cd $MRB_SOURCE/larsoft/releaseDB
51 2 Lynn Garren
* Make sure that larutils, larbatch, and mrb are removed from the cfg scripts.
52 2 Lynn Garren
** streamline this by creating a small "larsoft extras" distribution that can be optionally included.
53 2 Lynn Garren
* commit and push the change
54 2 Lynn Garren
55 4 Lynn Garren
h2(count). Making a patch release
56 2 Lynn Garren
57 2 Lynn Garren
* These notes presume that you do not have an existing directory
58 2 Lynn Garren
59 4 Lynn Garren
h3(count). MicroBooNE setup
60 2 Lynn Garren
61 2 Lynn Garren
* setup the latest release of larreltools
62 2 Lynn Garren
* startUboonePatchRel  <working_dir> <tag> <larsoft branch> [<uboone branch>]
63 2 Lynn Garren
** This script will create a directory named "tag" under the "working_dir".
64 2 Lynn Garren
** This script will clone the larsoft/larsoftobj on the named branch.
65 2 Lynn Garren
** The same branch name will be used when cloning the uboone_suite unless a second branch name is supplied.
66 2 Lynn Garren
** The directory structure will include a single source code directory with subdirectories for e17 and c2 debug and prof builds.
67 2 Lynn Garren
** Use "dogit branch" to double check that you are on the expected branch.
68 2 Lynn Garren
69 4 Lynn Garren
h3(count). Setup for other experiments
70 2 Lynn Garren
71 2 Lynn Garren
* Appropriate scripts will be supplied when necessary.
72 2 Lynn Garren
73 4 Lynn Garren
h3(count). Local development
74 2 Lynn Garren
75 2 Lynn Garren
* Before integrating any feature branches, make sure that the release builds as is and that the unit tests work.
76 2 Lynn Garren
** Make sure to build and test both e17 and c2.   Usually the prof build is sufficient for testing.  prof is preferred over debug, which sometimes masks problems.
77 2 Lynn Garren
<pre>
78 2 Lynn Garren
source local*/setup
79 2 Lynn Garren
cd $MRB_BUILDDIR
80 2 Lynn Garren
mrb t -jN
81 2 Lynn Garren
</pre>
82 2 Lynn Garren
* Next make changes and integrate feature branches
83 2 Lynn Garren
** Because feature branches may conflict or not have been thoroughly tested, we recommend merging one feature branch set at a time.  Build and test after each set.
84 2 Lynn Garren
85 4 Lynn Garren
h3(count). Update the package versions
86 2 Lynn Garren
87 2 Lynn Garren
* Do this once all changes have been made.
88 2 Lynn Garren
* ONLY update the version if there is a change.
89 2 Lynn Garren
* cd to each directory separately
90 2 Lynn Garren
* git diff LARSOFT_SUITE_vaa_bb_cc[_dd]
91 2 Lynn Garren
** You are comparing against the previous patch release in this set.
92 2 Lynn Garren
** If this is the first patch release in the set, compare against the original release.
93 2 Lynn Garren
** If there are no changes, move this package out of the way.
94 2 Lynn Garren
<pre>
95 2 Lynn Garren
cd $MRB_SOURCE
96 2 Lynn Garren
mkdir ../notag
97 2 Lynn Garren
mv xxx ../notag
98 2 Lynn Garren
mrb uc
99 2 Lynn Garren
</pre>
100 2 Lynn Garren
** Move all experiment code out of the way.  (_mv ub* ../notag/_ for MicroBooNE)
101 2 Lynn Garren
** Use "mrb uv" to update the version by one in the patch field.
102 2 Lynn Garren
*** There should be a script for this...
103 2 Lynn Garren
* You must update larsoftobj/bundle/CMakeLists.txt and larsoft/releaseDB/CMakeLists.txt by hand with the new versions.
104 2 Lynn Garren
** commit and push these changes
105 2 Lynn Garren
* DO NOT commit the changes to ups/product_deps
106 2 Lynn Garren
107 2 Lynn Garren
108 4 Lynn Garren
h3(count).  Tagging the release
109 2 Lynn Garren
110 2 Lynn Garren
* Do a final build and test with just the packages you expect to tag.  
111 2 Lynn Garren
** This step will also create the necessary build cfg files.
112 2 Lynn Garren
** If there are changes in both larsoft and larsoftobj, double check for consistency.
113 2 Lynn Garren
<pre>
114 2 Lynn Garren
cd $MRB_BUILDDIR/larsoft
115 2 Lynn Garren
diff releaseDB ../larsoftobj/bundle | grep diff
116 2 Lynn Garren
diff releaseDB ../larsoftobj/bundle | grep Only
117 2 Lynn Garren
</pre>
118 2 Lynn Garren
** If all is good, copy the cfg scripts to SciSoft
119 2 Lynn Garren
<pre>
120 2 Lynn Garren
cd $MRB_BUILDDIR/larsoft/releaseDB
121 2 Lynn Garren
copyToSciSoft lar*
122 2 Lynn Garren
</pre>
123 2 Lynn Garren
* Now tag
124 2 Lynn Garren
** tagPatchRel <existing branch> <new master tag>
125 2 Lynn Garren
** tagPatchRel will commit ups/product_deps with an appropriate message, make the tag, and push both tag and branch to origin.
126 2 Lynn Garren
** Use "dogit status" to doublecheck.
127 2 Lynn Garren
** *IMPORTANT*:  patch release tags are never merged with the master branch
128 2 Lynn Garren
129 4 Lynn Garren
h3(count).  Official build
130 2 Lynn Garren
131 2 Lynn Garren
* build on Jenkins
132 2 Lynn Garren
* Use special patch release build jobs.
133 2 Lynn Garren
** This avoids possible conflict with normal release builds.
134 1 Lynn Garren
* use copyFromJenkins to download the tarballs and release manifests from Jenkins
135 1 Lynn Garren
* use copyToSciSoft to upload the tarballs and release manifests to SciSoft
136 1 Lynn Garren
** This step requires extra privileges.
137 5 Lynn Garren
138 5 Lynn Garren
h3(count).  Release Notes and Cross Package Tag
139 5 Lynn Garren
140 2 Lynn Garren
* Once the build is complete for ALL variants (SLF and macOS), make the cross package tag.
141 5 Lynn Garren
** We wait because sometimes a problem is not found until building and we need to move a tag.
142 2 Lynn Garren
** This step requires redmine manager persmission for larsoft
143 2 Lynn Garren
<pre>
144 2 Lynn Garren
cp-lar-tag <larsoft tag> <larsoftobj tag>
145 2 Lynn Garren
</pre>
146 2 Lynn Garren
*** It would be nice to fix cp-lar-tag so it deduces the larsoftobj tag
147 2 Lynn Garren
* Make the release notes
148 2 Lynn Garren
** make-release-notes <working_directory> <tag> <previous tag>
149 2 Lynn Garren
*** This script will create a subdirectory named tag under the working_directory
150 2 Lynn Garren
** cd into the newly created tag directory
151 2 Lynn Garren
** cat ReleaseNotes-vaa_bb_cc_dd
152 2 Lynn Garren
*** The first line of this output is the entry in the [[LArSoft_release_list|release list]]
153 2 Lynn Garren
*** The remainder of the output is used to create the release note itself.
154 1 Lynn Garren
**** copy and edit the top part if appropriate.
155 1 Lynn Garren
**** Do not edit anything under the "Change List" section.
156 1 Lynn Garren
**** It is important to use cat so that there are no unexpected line breaks.
157 5 Lynn Garren
158 5 Lynn Garren
h3(count). Upload to cvmfs
159 5 Lynn Garren
160 5 Lynn Garren
* This step requires login privileges.
161 5 Lynn Garren
* ssh cvmfslarsoft@oasiscfs.fnal.gov
162 5 Lynn Garren
* The README provides an overview of the necessary steps.
163 5 Lynn Garren
** start a transaction
164 5 Lynn Garren
** install using installBundle.sh
165 5 Lynn Garren
** publish
166 5 Lynn Garren
* installBundle.sh larsoft vaa_bb_cc_zz sNN-eY
167 5 Lynn Garren
** Note that if you call installBundle.sh with sNN-e17, it will install both e17 and c2 variants.
168 5 Lynn Garren
** The script will attempt to install the release for all supported platforms and clean up after itself.
169 5 Lynn Garren
** If some expected distributions are not available, then manifests will be left in the .working subdirectory. Check and remove them if necessary.