LArIAT code releases

Most of this is shamelessly borrowed (read: stolen) from uboonecode.


Preparing a release from the develop branch (git flow release method)

Set up the necessary Unix Product Support (UPS) products:

source /grid/fermiapp/larsoft/products/setup
export PRODUCTS=/grid/fermiapp/products/lariat:${PRODUCTS}:/grid/fermiapp/products/common/db
setup git
setup gitflow
setup mrb
setup ninja v1_8_2

Create a mrb development area:

export MRB_PROJECT=larsoft
cd <path>  # choose your own path
mrb newDev -v develop -q e15:prof
source <path>/localProducts*/setup

Check out lariatsoft, lariatutil, and lariatfragments:

mrb g lariatsoft
mrb g lariatutil
mrb g lariatfragments

Make sure your local copies of develop and master branches are up to date with the main repository. Do this for each checked out package.

cd $MRB_SOURCE/<package>  # <package> = lariatsoft, lariatfragments, lariatutil
git checkout develop
git pull
git checkout master
git pull

Use git flow to create a release branch in your local repository. Git flow always creates the release branch from the head of develop.

git flow release start <version>

This command creates a new branch off of the head of develop called release/<version>. After this command, the newly created release branch will be the checked out branch. At this point, you can make further changes to your release branch, including the version number of the package being tagged and version numbers of dependent products (e.g. larbatch and cetbuildtools). In particular, you may want to edit the following files.

  • ups/product_deps

After making all updates, it is a good idea to do a full test build, including tests.

mrb z && mrbsetenv
mrb i --generator ninja
mrb t --generator ninja

See the Continuous integration (CI) test section below for instructions on running the continuous integration (CI) test locally.

When you are done updating the release branch, commit updates to your local repository for each package.

cd $MRB_SOURCE/<package>  # <package> = lariatsoft, lariatfragments, lariatutil
git commit -a -m <version>

Use git flow to "finish" the release branch.

git flow release finish -m <version>
This git flow command does the following actions.
  • Merges release branch to master.
  • Merges release branch to develop.
  • Creates a tag on the master branch.
  • Deletes the release branch.
  • Checks out the develop branch.

Note that when you merge the release branch into master, you are also merging any updates originally from develop into master.

Now push everything up to main repository.

git push origin develop master
git push --tags

Preparing a hotfix release (git flow hotfix method)

Prepare your development area as in the previous section, up to the "git flow release start" command. If you are patching a release that corresponds to the current head of the master branch (i.e. the most recent tag), start your hotfix branch using the following command.

git flow hotfix start <new version>

If you want to patch an older tag, you must first create a new branch off of that tag.

git checkout -b <bug_fix_branch> <old version>
git flow hotfix start <new version> <bug_fix_branch>

Either command creates a new branch called hotfix/<new version>. Make updates and test your hotfix branch as in the previous section. When you are done, use the following command to finish the hotfix.

git flow hotfix finish

This command merges the hotfix branch to master (or bug fix) and develop branches, and creates a new tag on the master or bug fix branch. Then push your updates up to the main repository.

git push origin develop [master|<bug_fix_branch>]
git push --tags

The main difference between "git flow release" and "git flow hotfix" is that the former works off of develop, and the latter works off of master (or bug fix branch).

Continuous integration (CI) test

setup cigetcert
cigetcert -s
voms-proxy-init -noregen -rfc -voms fermilab:/fermilab/lariat/Role=Analysis
setup lar_ci
setup lariatsoft $LARIATSOFT_VERSION -q $MRB_QUALS
cd <path>  # choose your own path
test_runner -v -s quick_test_lariatsoft

Building releases using Jenkins

This section explains how to build a release using the Jenkins build machine. The instructions include how to upload releases to the ups products server and install in the Fermilab ups products area /grid/fermiapp.

Scisoft scripts

The method described in this section makes use of several scripts that can be downloaded from, namely:

The above scripts are not part of any ups product, but rather must be downloaded directly from scisoft and made executable. If you use these scripts frequently, you may want to keep copies in a personal script location that is on your execution path.

Jenkins projects, OSes, build types, and qualifiers

Jenkins builds are organized by a multidimensional matrix consisting of dimensions such as those summarized in the following table.

Project OS flavor Build type Qualifiers
lariat-release-build slf6 debug e15
lariat-release-build slf6 prof e15
lariat-release-build slf7 debug e15
lariat-release-build slf7 prof e15
lariat-release-build-mac d15 (a.k.a. macOS 10.11 a.k.a. El Capitan) debug e15
lariat-release-build-mac d15 (a.k.a. macOS 10.11 a.k.a. El Capitan) prof e15
lariat-release-build-mac d16 (a.k.a. macOS 10.12 a.k.a. Sierra) debug e15
lariat-release-build-mac d16 (a.k.a. macOS 10.12 a.k.a. Sierra) prof e15

Each row of the above table corresponds to a different binary flavor.

Build release

Here are step-by-step instructions for controlling the build via the Jenkins web interface.

  1. Open the Jenkins home page (Jenkins dashboard) Make sure you are authenticated (user name appears in upper right).
  2. Click on tab "lariat".
  3. Click on a project "lariat-release-build."
  4. Click on "Configure." Scroll down to the parameters section and make any desired updates to default values (package versions and qualifiers).
  5. Click on "Build with Parameters." You will be given another chance to update parameters. If you edited the default values of the parameters in the previous step, the listed values should already be correct. Click "Build."
  6. Wait for build to complete. Progress of the build can be monitored using the progress bar in the Build History box. While the build is progressing, status icons on the project page will blink. When build is complete, status icons will stop blinking. Make sure that each binary flavor built successfully (status icon displayed as green ball).

Upload release to scisoft

Do this step after a successful Jenkins build. Make sure you have access to scisoft scripts copyFromJenkins and copyToSciSoft on your execution path.

  1. Go to an empty directory on any machine where you have Kerberized scp access to (lariatgpvmXX login machines are fine). You will assemble in this directory all files that will be uploaded to
  2. Fetch results of Jenkins build using copyFromJenkins. This script is normally invoked with a single argument specifying the Jenkins project name.
copyFromJenkins lariat-release-build

This command will fetch Jenkins build "artifacts," including tarballs (*.tar.bz2 or *.tar.gz), manifests (*.txt), and html files. There should be one html file, one manifest for each binary flavor, and one tarball for each binary flavor of each ups product.

  1. Upload all files to scisoft using copyToSciSoft. Specify all files that you want to upload on the command line, which should be all of the files your current directory.
copyToSciSoft *

Script copyToSciSoft decides where to copy files based on the file name and file type. The files returned by Jenkins (tarballs, manifests, and release html file) all follow naming conventions that are understood by copyToSciSoft.

copyToSciSoft is written such that it will not replace an existing file with the same name. If you make a mistake and need to replace an exiting file on, ssh in to and cd to /nasroot/SciSoft. From there you can make arbitrary edits of files being served by, including deleting them.

Install release in /grid/fermiapp

Log in to the lariatsoft account on any lariatgpvmXX machine. You will need access the scisoft script pullProducts. Use the copyFromJenkins script from above to fetch the Jenkins build artifacts. This script should already be on your execution path when you log in as lariatsoft.

ssh -l lariatsoft

Go to an empty directory and fetch the Jenkins build artifacts for SLF6 and SLF7:

cd <path>  # choose your own path
copyFromJenkins -m slf6 lariat-release-build
copyFromJenkins -m slf7 lariat-release-build

Untar the packages to the /grid/fermiapp/products/lariat directory:

cd /grid/fermiapp/products/lariat
tar xvf <path>/<package>-<version>-slf6-x86_64-e15-debug.tar.bz2  # <package> = lariatsoft, lariatfragments, lariatutil
tar xvf <path>/<package>-<version>-slf6-x86_64-e15-prof.tar.bz2   # <package> = lariatsoft, lariatfragments, lariatutil
tar xvf <path>/<package>-<version>-slf7-x86_64-e15-debug.tar.bz2  # <package> = lariatsoft, lariatfragments, lariatutil
tar xvf <path>/<package>-<version>-slf7-x86_64-e15-prof.tar.bz2   # <package> = lariatsoft, lariatfragments, lariatutil


Logging in to the Fermilab CVMFS server

You can log in to from any lariatgpvmXX node:

ssh -l cvmfslariat

Updating CVMFS repository contents

After logging in to, run the following command to add the LArIAT software packages to the CVMFS repository:

lariat-sync <package>/<version>* <package>  # <package> = lariatsoft, lariatfragments, lariatutil

Manually publishing files to CVMFS repository

Log in to

ssh -l cvmfslariat

Begin CVMFS transaction:

cvmfs_server transaction

scp/rsync files into CVMFS directory:

scp -r /cvmfs/

Move to a directory other than /cvmfs/

cd ~

Publish transaction:

cvmfs_server publish

Useful mailing lists

Here is a list of useful mailing lists:




To subscribe, send an email to with the subject line blank and the following body:

subscribe <mailing list name> <first name> <last name>