Notes for NuTools release managers¶
- Table of contents
- Notes for NuTools release managers
Updating one package¶
- If you only need a new version of a single package, such as nugen, then simply checkout that package, edit, tag and build.
- If you want a complete nulite distribution with the new package, then you would also checkout nutools and follow the instructions for updating the nu suite.
The nu suite¶
The nu suite consists of nutools, nuevdb, nug4, nugen, nurandom, and nusimdata.
Branch names¶
- Branches for NOvA are of the form nova_v3_01_br
- Branches for LArSoft are of the form lar_v2_27_br
- Because LArSoft moves ahead with art more frequently than NOvA, most larsoft distributions are made off the head of develop.
Tools¶
- mrb v3_04_04 or later
- This version adds the nu_suite gitCheckout option and fixes some bugs in mrb uv.
- buildFW (used by the Jenkins build)
- copyFromJenkins
- copyToSciSoft
- pullProducts
- pullPackage
- larreltools v1_08_04 or later
- This package was created for LArSoft, but contains common tools.
- dogit
- startNuLiteRel
- manageNuLite
Updating the entire nutools suite from the head of develop¶
These instructions presume that the work is done on scisoftbuild01.fnal.gov. These instructions also presume that the head of develop will be used.
update "third party" products¶
Before building the nu suite, new releases of "third party" products need to be available. Our definition of "third party products" is any product that does not use cetbuildtools or cetmodules.
This example updates the nu suite to use art 3.04.00. python3 is the default. This means that the py3 qualifier is no longer used. Instead, the py2 qualifier is used to specify a build with python v2_7_15a. Because of this change, root has been updated to v6_18_04c. Many products in the nulite distribution will need new releases with updated table files.
- genie v3_00_06e
- dk2nudata v1_08_00f
- dk2nugenie v1_08_00h
setup the working environment¶
source /products/setup setup mrb setup larreltools export PATH=$LARRELTOOLS_DIR/expert:${PATH} export MRB_PROJECT=nu
make a working directory and checkout the code¶
Use the startNuLiteRel script.
Usage: startNuLiteRel <working_dir> <new tag> <base qualifier> This script will do the following: mkdir <working_dir>/<tag> run newDev git clone nulite repositories from redmine The base qualifier should be one of e17, e19, c7 etc.
For instance:
startNuLiteRel `pwd` v3_06_00 e19 startNuLiteRel `pwd` v3_06_00 c7
update, build, and test the code¶
Create a temporary release branch. This workflow mimics, but does not use, gitflow. This allows us more flexibility when dealing with multiple repositories.
source v3_06_00/e19p/localProducts_nu_v3_06_00_e19_prof/setup cd $MRB_SOURCE manageNuLite start $MRB_PROJECT_VERSION
Now, we update the code. This example updates the head of the nutools suite to use art 3.04.
mrb uv canvas_root_io v1_05_00 mrb uv art v3_04_00 mrb uv art_root_io v1_02_00 mrb uv gallery v1_14_00 mrb uv toyExperiment v0_05_00 mrb uv ifdh_art v2_10_01
In this case, we use mrb cq (changeQualifier) from py3 to py2 and also update the version_set.
mrb uv version_set s94 mrb cq py3 py2
Make sure the required products are available. We install the critic distribution in /products on scisoftbuild01. If there are local builds of other products, make sure they are installed in any private product directory. Add symbolic links to the product in $MRB_INSTALL. If you change PRODUCTS to point to your product directory, make sure it does not contain test versions of products.
cd $MRB_INSTALL ln -s /home/garren/scratch/products/genie etc.
Build and test. Iterate as necessary until everything builds cleanly.
cd $MRB_BUILDDIR mrb z (when necessary) mrbsetenv mrb t -jN
Tagging the release¶
First update the version. In this case, the art release and accompanying qualifiers have changed. The convention is to update the minor version of the packages in the nu suite.
updateVersion minor
It's wise to have a look at each package and make sure the changes are what you expect.
Updating part of the nutools suite.¶
update required products¶
Before building the nu suite, new releases of required products need to be available.
This example updates the nu suite to use ifdhc v2_5_4, libwda v2_27_0, and ifdh_art v2_10_02.
python3 is the default. This means that the py3 qualifier is no longer used. Instead, the py2 qualifier is used to specify a build with python v2_7_15a.
setup the working environment¶
source /products/setup setup mrb setup larreltools export PATH=$LARRELTOOLS_DIR/expert:${PATH} export MRB_PROJECT=nu
make a working directory and checkout the code¶
Use the startNuLiteRel script.
Usage: startNuLiteRel <working_dir> <new tag> <base qualifier> This script will do the following: mkdir <working_dir>/<tag> run newDev git clone nulite repositories from redmine The base qualifier should be one of e17, e19, c7 etc.
For instance:
startNuLiteRel `pwd` v3_06_00a e19 startNuLiteRel `pwd` v3_06_00a c7
update the code¶
We use mrb uv to update all products that are part of the nulite distribution.
source v3_06_00a/e19p/local*/setup cd $MRB_SOURCE manageNuLite start v3_06_00a (This step creates a working branch named release/v3_06_00a) mrb uv ifdhc v2_5_4 mrb uv ifdch_config v2_5_4 mrb uv ifbeam v2_4_2 mrb uv nucondb v2_4_2 mrb uv libwda v2_27_0 mrb uv ifdh_art v2_10_02
Check for packages that have no changes and move them out of the way. This list will vary depending on which dependencies have been updated.
cd $MRB_SOURCE dogit status mkdir ../nobuild mv nusimdata ../nobuild/ mv nurandom ../nobuild/ mv nug4 ../nobuild/ mrb uc
build and test the code¶
Iterate as necessary until everything builds cleanly.
cd $MRB_BUILDDIR mrb z (if the directory is not empty) mrbsetenv mrb t -jN
update package versions¶
In this example, we will just update the micro version. Note that manageNuLite update will update versions in ups/product_deps and in the nutools/bundle directory. The script automatically calculates the next release number in the appropriate series (major, minor, or micro).
manageNuLite update micro
special instructions for nutools¶
nutools contains the build config templates. These templates must be updated, but there is no reason to make a new release of nutools unless something in nutools itself has changed. If necessary, reset the nutools release to the previous version and update the nulite (distribution) release with an alphanumeric character. In this example, the only changes in nutools will be to nutools/bundle/CMakeLists.txt. The nutools/bundle directory contains the distribution templates.
cd $MRB_SOURCE/nutools git diff ups/product_deps mrb uv nutools v3_06_00 git diff ups/product_deps (should show no changes) mrb uv nulite v3_06_00a
Final build and test¶
Make one more test build to make sure there are no problems. This build will also generate the scripts needed to build the nulite distribution.
cd $MRB_BUILDDIR mrb z (if the directory is not empty) mrbsetenv mrb t -jN
Copy the nulite distribution scripts to SciSoft.
cd $MRB_BUILDDIR/nutools/bundle copyToSciSoft nulite*
Tag the updated packages¶
manageNuLite tag will do the following:- commit changes in ups/product_deps
- merge the release branch with both master and develop
- make the tag on the master branch
- push master, develop, and the tag to the remote repository
- if the nutools release is unchanged, manageNuLite tag will simply merge the release branch with develop and push to origin
- If any file other than ups/product_deps has changed, this step will fail.
- You can rerun this command as many times as necessary without problem.
cd $MRB_SOURCE manageNuLite tag v3_06_00a
Updating the nutools suite on a branch¶
- Branches for NOvA are of the form nova_v3_01_br
- Branches for LArSoft are of the form lar_v2_27_br
setup the working environment¶
source /cvmfs/nova.opensciencegrid.org/externals/setup export PRODUCTS=$PRODUCTS:/cvmfs/larsoft.opensciencegrid.org/products setup mrb setup larreltools export PATH=$LARRELTOOLS_DIR/expert:${PATH} export MRB_PROJECT=nu
make a working directory and checkout the code¶
makeNuLiteWorkDir `pwd` v3_01_04 e17 makeNuLiteWorkDir `pwd` v3_01_04 c2
Update nutools for MicroBooNE production releases v08_05_00_NN¶
The MicroBooNE v08_05_00_NN production releases are built with nutools v2_27_XX. This was before the nutools v3 split.
Before building nutools, new releases of "third party" products need to be available.
setup the working environment¶
source /products/setup setup mrb export MRB_PROJECT=nu
make a working directory and checkout the code¶
Since this release uses the old monolithic nutools, it is not necessary to use mrb. However, we recommend using mrb so you can update product versions with the "mrb uv" command.
mkdir <working_dir>/<new tag> cd <working_dir>/<new tag> mrb newDev -v <new tag> -q e17:prof source <working_dir>/<new tag>/localProducts*/setup mrb g -b lar_v2_27_br nutools Verify that you are on the lar_v2_27_br branch of nutools.
update, build, and test the code¶
Now, we update the code. This example updates the nutools branch to use a newer genie and accompanying dk2nudata.
mrb uv genie v3_00_04_ub1 mrb uv dk2nugenie v01_08_00_ub1 mrb uv genie_phyopt v3_00_04 (only if needed) mrb uv genie_xsec v3_00_04a (only if needed)
Make sure the required products are available. If there are local builds of some products, make sure they are installed in a private product directory. Add symbolic links to the product in $MRB_INSTALL.
cd $MRB_INSTALL ln -s /home/garren/scratch/products/genie ln -s /home/garren/scratch/products/dk2nugenie
Build and test. Iterate as necessary until everything builds cleanly.
cd $MRB_BUILDDIR mrb z (when necessary) mrbsetenv mrb t -jN
Update version¶
Make sure you update the nutools version.
mrb uv nutools <new tag> mrb uv nulite <new tag>
Final build and test¶
Make one more test build to make sure there are no problems. This build will also generate the scripts needed to build the nulite distribution.
cd $MRB_BUILDDIR mrb z (if the directory is not empty) mrbsetenv mrb t -jN
Copy the nulite distribution scripts to SciSoft. It is advisable to compare them to the scripts for the previous distribution before uploading.
cd $MRB_BUILDDIR/nutools/bundle copyToSciSoft nulite*
Tag¶
Make sure you commit all changes in nutools. Once that is done, generate the tag and update.
cd $MRB_SOURCE/nutools git commit -m"..." file(s) git tag -a -m"..." <new tag> git push origin lar_v2_27_br <new tag>
Build the nulite distribution¶
There are several Jenkins jobs that build the nulite distribution.
Choose "Build with Parameters" and edit NUVER appropriately.
- Use build-nulite for the head nutools.
- Use build-nulite-s85 for NOvA.
- Use build-nulite-s78 to make a distribution for the MicroBooNE v08_05_00_xx larsoft patch releases.
When the build finishes¶
Sometimes a problem shows up the the build that was not caught during testing. If this happens, make the fix and rebuild. It is OK to move a tag for code in redmine if there is no build yet on SciSoft. If there is already a build on SciSoft, a new tag is required.
When the job finishes successfully, the manifests and tarballs will be in "Last Successful Artifacts" for each matrix entry. Click on the green ball to see them. Since it is tedious at best to copy directly from Jenkins, use copyFromJenkins. You can do this on any machine, but I prefer to use scisoftportal. Make sure you download to an empty directory so you can use the wildcard.
The list of qualifiers will be defined by the Jenkins build job. This example is for build-nulite with s94.
ssh scisoftportal.fnal.gov mkdir tmp cd tmp /nasroot/SciSoft/bundles/tools/copyFromJenkins -N -q s94-c7 -q s94-c7-py2 -q s94-e19 -q s94-e19-py2 build-nulite /nasroot/SciSoft/bundles/tools/copyToSciSoft *
Notification¶
Notify the appropriate people that the build is complete and available on SciSoft.- If the build was made in response to a redmine ticket, update the ticket.
- If the build is needed for an upcoming larsoft release, send email to scisoft-team@fnal.gov.