Project

General

Profile

Notes for NuTools release managers » History » Version 32

« Previous - Version 32/50 (diff) - Next » - Current version
Lynn Garren, 02/04/2020 02:08 PM


Notes for NuTools release managers

Updating one package

  • If you only need a new version of a single package, such as nugen, then simply checkout that package, edit, tag and build.
  • If you want a complete nulite distribution with the new package, then you would also checkout nutools and follow the instructions for updating the nu suite.

The nu suite

The nu suite consists of nutools, nuevdb, nug4, nugen, nurandom, and nusimdata.

Branch names

  • Branches for NOvA are of the form nova_v3_01_br
  • Branches for LArSoft are of the form lar_v2_27_br
  • Because LArSoft moves ahead with art more frequently than NOvA, most larsoft distributions are made off the head of develop.

Tools

  • mrb v3_04_04 or later
    • This version adds the nu_suite gitCheckout option and fixes some bugs in mrb uv.
  • buildFW (used by the Jenkins build)
  • copyFromJenkins
  • copyToSciSoft
  • pullProducts
  • pullPackage
  • larreltools v1_08_00 or later
    • This package was created for LArSoft, but contains common tools.
    • dogit
    • startNuLiteRel
    • updateVersion

Updating the entire nutools suite from the head of develop

These instructions presume that the work is done on scisoftbuild01.fnal.gov. These instructions also presume that the head of develop will be used.

update "third party" products

Before building the nu suite, new releases of "third party" products need to be available. Our definition of "third party products" is any product that does not use cetbuildtools or cetmodules.

This example updates the nu suite to use art 3.04.00. python3 is the default. This means that the py3 qualifier is no longer used. Instead, the py2 qualifier is used to specify a build with python v2_7_15a. Because of this change, root has been updated to v6_18_04c. Many products in the nulite distribution will need new releases with updated table files.

  • genie v3_00_06e
  • dk2nudata v1_08_00f
  • dk2nugenie v1_08_00h

setup the working environment

source /products/setup
setup mrb
setup larreltools
export PATH=$LARRELTOOLS_DIR/expert:${PATH}
export MRB_PROJECT=nu

make a working directory and checkout the code

Use the startNuLiteRel script.

Usage: startNuLiteRel <working_dir> <new tag> <base qualifier>
   This script will do the following:
       mkdir <working_dir>/<tag>
       run newDev
       git clone nulite repositories from redmine
   The base qualifier should be one of e17, e19, c7 etc.

For instance:
startNuLiteRel `pwd` v3_06_00 e19
startNuLiteRel `pwd` v3_06_00 c7

update, build, and test the code

Create a temporary release branch. This workflow mimics, but does not use, gitflow. This allows us more flexibility when dealing with multiple repositories.

source v3_06_00/e19p/localProducts_nu_v3_06_00_e19_prof/setup
cd $MRB_SOURCE
manageNuLite start $MRB_PROJECT_VERSION

Now, we update the code. This example updates the head of the nutools suite to use art 3.04.
mrb uv canvas_root_io v1_05_00
mrb uv art v3_04_00
mrb uv art_root_io v1_02_00
mrb uv gallery v1_14_00
mrb uv toyExperiment v0_05_00
mrb uv ifdh_art v2_10_01

In this case, we use mrb cq (changeQualifier) from py3 to py2 and also update the version_set.
mrb uv version_set s94
mrb cq py3 py2

Make sure the required products are available. We install the critic distribution in /products on scisoftbuild01. If there are local builds of other products, make sure they are installed in any private product directory. Add symbolic links to the product in $MRB_INSTALL. If you change PRODUCTS to point to your product directory, make sure it does not contain test versions of products.
cd $MRB_INSTALL
ln -s /home/garren/scratch/products/genie
etc.

Build and test. Iterate as necessary until everything builds cleanly.
cd $MRB_BUILDDIR
mrb z (when necessary)
mrbsetenv
mrb t -jN

Tagging the release

First update the version. In this case, the art release and accompanying qualifiers have changed. The convention is to update the minor version of the packages in the nu suite.

updateVersion minor

It's wise to have a look at each package and make sure the changes are what you expect.

Updating part of the nutools suite.

update required products

Before building the nu suite, new releases of required products need to be available.

This example updates the nu suite to use ifdhc v2_5_4, libwda v2_27_0, and ifdh_art v2_10_02.

python3 is the default. This means that the py3 qualifier is no longer used. Instead, the py2 qualifier is used to specify a build with python v2_7_15a.

setup the working environment

source /products/setup
setup mrb
setup larreltools
export PATH=$LARRELTOOLS_DIR/expert:${PATH}
export MRB_PROJECT=nu

make a working directory and checkout the code

Use the startNuLiteRel script.

Usage: startNuLiteRel <working_dir> <new tag> <base qualifier>
   This script will do the following:
       mkdir <working_dir>/<tag>
       run newDev
       git clone nulite repositories from redmine
   The base qualifier should be one of e17, e19, c7 etc.

For instance:
startNuLiteRel `pwd` v3_06_00a e19
startNuLiteRel `pwd` v3_06_00a c7

update the code

source v3_06_01/e19p/local*/setup
cd $MRB_SOURCE
manageNuLite start v3_06_00a
(This step creates a working branch named release/v3_06_00a)
mrb uv ifdhc v2_5_4 (unnecessary)
mrb uv libwda v2_27_0
mrb uv ifdh_art v2_10_02

build and test the code

Iterate as necessary until everything builds cleanly.

cd $MRB_BUILDDIR
mrb z (if the directory is not empty)
mrbsetenv
mrb t -jN

update package versions

In this example, we will just update the micro version. Note that manageNuLite update will update versions in ups/product_deps and in the nutools/bundle directory. The script automatically calculates the next release number in the appropriate series (major, minor, or micro).

manageNuLite update micro

special instructions for nutools

nutools contains the build config templates. These templates must be updated, but there is no reason to make a new release of nutools unless something in nutools itself has changed. Reset the nutools release to the previous version and update the nulite (distribution) release with an alphanumeric character. In this example, the only changes in nutools will be to nutools/bundle/CMakeLists.txt. The nutools/bundle directory contains the distribution templates.

cd $MRB_SOURCE/nutools
git diff ups/product_deps
mrb uv nutools v3_06_00
git diff ups/product_deps (should show no changes)
mrb uv nulite v3_06_00a

Final build and test

Make one more test build to make sure there are no problems. This build will also generate the scripts needed to build the nulite distribution.

cd $MRB_BUILDDIR
mrb z (if the directory is not empty)
mrbsetenv
mrb t -jN

Copy the nulite distribution scripts to SciSoft.
cd $MRB_BUILDDIR/nutools/bundle
copyToSciSoft nulite*

Push changes for nutools

Tag the updated packages

Updating the nutools suite on a branch

All NOvA releases are tagged and built off a special branch.

setup the working environment

source /cvmfs/nova.opensciencegrid.org/externals/setup
export PRODUCTS=$PRODUCTS:/cvmfs/larsoft.opensciencegrid.org/products
setup mrb
setup larreltools
export PATH=$LARRELTOOLS_DIR/expert:${PATH}
export MRB_PROJECT=nu

make a working directory and checkout the code

makeNuLiteWorkDir `pwd` v3_01_04 e17
makeNuLiteWorkDir `pwd` v3_01_04 c2

just saving for reference

  • If you want a complete nulite distribution with the new package, then you would also checkout nutools and follow the instructions for updating the nu suite.
    • Edit the package version and the nudist version in nutools/bundle/CMakeLists.txt.
    • The nudist version must be the same as the nutools version, but with an alphabet character added to the end.
      • For instance, v3_01_03 would become v3_01_03a, v3_01_03b would become v3_01_03c, etc.
    • Build nutools and use the generated nutools/bundle/nu* files. (See below for more information.)