Setting up the Offline Software


If you are working on a computer not at Fermilab, you need to download LArSoft and the related packages.

Tagged Releases

version qualifers
v06_31_00 e10:prof

Your Development Release

LArIATSoft makes use of the mrb tool used by LArSoft to develop software. Thus the steps to develop software for LArIAT are very similar to the steps to develop LArSoft.

As of the release of cetbuildtools v4_08_00 there is the option for using the ninja build file generation system for faster builds rather than the usual make system. One has to explicitly setup and call ninja and the instructions below assume that everyone will want to do so. After sourcing the setup file in the /grid/fermiapp/products/larsoft area, you can setup ninja by doing

setup ninja v1_6_0b

The example bash function in Option 1 below does this for you. For Options 2 and 3 the instructions have been updated to do the setup explicitly, but one can also add that setup to a .bash_profile script so as not to have to do it explicitly after each log in.

Option 1: Establishing a distinct LArIATSoft development area from your LArSoft code development area

Setting up a development area can be accomplished by adding the following function to your .bash_profile on the lartiatgpvm nodes:

    # first source the larsoft product area and setup mrb, git and git flow
    # also add the LArIAT products area to the $PRODUCT variable
    source /grid/fermiapp/larsoft/products/setup
    export PRODUCTS=/grid/fermiapp/products/lariat:${PRODUCTS}:/grid/fermiapp/products/common/db
    setup git
    setup gitflow
    setup mrb
    setup ninja v1_6_0b

    # define the necessary mrb environmental variables and some useful LArIAT related ones
    export MRB_PROJECT=lariatsoft

    #!!!!!!!!----------Change the next line to point to where you want to install lariatsoft-------!!!!!!!!!!!!
    export LARIATDEVDIR=$HOME/lariat
    if [ -d "$LARIATDEVDIR" ]; then
        cd ${LARIATDEVDIR}
        echo "LARIATDEVDIR is set to ${LARIATDEVDIR}, which does not exist. " 
        echo "Please set LARIATDEVDIR to the location where you wish to develop lariatsoft" 
        echo "and edit the develop_lariat function in your .bash_profile appropriately" 
        return 0

    # check if any arguments were given to the function.
    # the arguments are
    # 1: version in format of either "develop" or XX_YY_ZZ
    # 2: Base qualifier, ie eX, depends on the version of art
    # 3: Build qualifier, either debug or prof
    if [ -z $1 ] ; then

    # define more variables for convenience based on the input arguments
    export LOCAL_LARIAT=${version}
    export LOCAL_PRODUCTDIR=localProducts_${MRB_PROJECT}_${version}_${ebase}_${build}

    # check that the local lariat area is there, if not,
    # this is a new version to check out so do that
    if [ -d "$LOCAL_LARIAT" ]; then
        cd ${LOCAL_LARIAT}
        source ${LOCAL_PRODUCTDIR}/setup
        # comment out the next 3 lines if you do not want to update your copy of the
        # repository with each log in
        echo "-------------------Updating git------------------" 
        cd srcs/lariatsoft
        git pull origin develop
        echo "-------------------------------------------------" 
        # uncomment the following 2 lines if you want to set up the ups product
        # with each log in
        #echo "-------------Setting up UPS product--------------" 
        #setup lariatsoft ${LARIATSOFT_VERSION} -q ${ebase}:${build}
        mkdir ${LOCAL_LARIAT}
        cd ${LOCAL_LARIAT}
        mrb newDev -v ${version} -q ${ebase}:${build}
        source ${LOCAL_PRODUCTDIR}/setup
        # Comment out the lines to check out the repository if you don't want to
        # do that with each log in
        echo "-----------Checking Repository Out-----------------" 
        cd srcs/
        mrb gitCheckout ssh://
        # Comment out the lines to build the code if you don't want to
        # do that with each log in
        echo "---------------Building Code----------------------" 
        ### !!!!!!!! Change slf6 to slf5 if using that OS !!!!!!!!! ###
        cd ${LARIATDEVDIR}/${LOCAL_LARIAT}/build_slf6.x86_64
        mrb i --generator ninja

    echo "-------------------------------------------------" 
    echo "Ready to develop LArIATSoft version ${LARIATSOFT_VERSION} in ${LARIATDEVDIR}/${LOCAL_LARIAT}" 
    echo " " 
    echo "When you want to run a job, please setup the ups product by doing" 
    echo "setup lariatsoft ${LARIATSOFT_VERSION} -q ${ebase}:${build}" 
    echo "-------------------------------------------------" 

The function is called by doing

$ develop_lariat 

To setup a default version named "develop" with the "eX" and "debug" qualifiers do

$ setup lariatsoft develop -q eX:debug

NB you should change the "X" in "eX" above to be the latest value. The "eX" qualifier is the same as the version of LArSoft being used.

If you want to setup a specific tagged version of lariatsoft with the version format XX_YY_ZZ, base qualifier <base qualifer>, i.e. eX, and build qualifier <build qualifier>, i.e. debug, do

$ develop_lariat XX_YY_ZZ <base qualifier> <build qualifier>

Note, that in both examples you are not setting up larsoft explicitly. That is done automatically for you when you source the ${LOCAL_PRODUCTDIR}/setup file. That step also sets the FHICL_FILE_PATH variable that points to the ${LOCAL_PRODUCTDIR}/lariat/<version>/fcl directory where all lariat related .fcl files are kept. If you have private .fcl files to use, they will be installed there automatically when you build and install the code if your package containing those files is in the srcs/lariat directory and you have correctly defined your CMakeLists.txt file for that package.

Also note, that you do not need to explicitly source either the LArSoft products area "setup" file or your local LArIATSoft product area "setup" file as those steps are done by the script.

Option 2: Developing LArIATSoft within your LArSoft development area

To add LArIATSoft to your LArSoft development area, you should first setup your LArSoft development environment. Then

setup ninja v1_6_0b # could put this line into a .bash_profile script
mrb g lariatsoft
mrb uc
mrb z
mrb i --generator ninja

You can add the above lines to whatever function you have in your .bash_profile to setup the LArSoft environment. You need only do the "mrb g" step once, in the future you can do

cd $MRB_SOURCE/lariatsoft
git pull origin develop

to update your LArIATSoft code.

Option 3: Developing LArSoft within your LArIATSoft development area

To add LArSoft to your LArIATSoft development area, you should first setup your LArIATSoft development environment. Then

setup ninja v1_6_0b # could put this line into a .bash_profile script
mrb g <insert larsoft module you want to develop here, i.e. larcore>
mrb uc
mrb z
mrb i --generator ninja

You can add the above lines to whatever function you have in your .bash_profile to setup the LArSoft environment. You need only do the "mrb g" step once, in the future you can do

cd $MRB_SOURCE/<larsoft module name>
git pull origin develop

to update your LArSoft code.

When a new LArSoft release is available and you want to develop against the new LArSoft release

  • Relogin a lariatgpvm node, do the following
    source /grid/fermiapp/lariat/
    setup larsoft vxx_yy_zz -q e10:prof #set up the latest larsoft release
    cd larsoft_mydev
    mrb newDev -p #this creates a new localProducts_XXXX directory using the new larsoft release and the existing srcs directory
    source localProducts_XXXX/setup  #make sure to use the new localProducts directory, it's better to delete the old directory
    cd srcs/lariatsoft
    git checkout develop
    git pull
  • If you are working on a feature branch:
    cd srcs/lariatsoft
    git checkout develop
    git pull
    git checkout feature/<my feature branch>
    git merge develop
  • Update all other repositories (e.g. larreco, lariatutil if you have those checked out in srcs) in the same way as updating lariatsoft.
  • Make a clean build
    mrb z
    mrb i --generator ninja -j4 #rebuild

Fast builds for code debugging

If you are making minor changes to the code, that is not changing the contents of the CMakeLists.txt files in your development area, you can avoid having mrb regenerate all the necessary CMake related files by simply running ninja (or make). For example


will build the code but not install it. That is useful if you want to test that the code compiles. If you want to also then run a job with your changes do

ninja install

which will install the compiled libraries into your local products area. You may substitute "make" for "ninja" in the above, but it will be a slower compilation process.

Developing LArIAT-Online code in the same area as LArIATSoft

Don't do it. LArIAT-Online has different dependencies from LArIATSoft and so it makes trying to develop them in the same area very hard to do. There is no certainty that the dependencies will keep pace with each other either. The good news is that we only need the data product libraries, which don't bring in dependencies with them.

To develop LArIAT-Online code, follow these instructions.

Developing in OS X Yosemite

NB The following instructions work for OS X 10.10 (Yosemite).

ART provides prebuilt ups products for most of the necessary external products to build lariatsoft on OS X. The full set of products will take up about 10GB of space on your hard drive.

1. Make a directory to contain the external products and then pull the products necessary for the lariatsoft bundle

mkdir /path/to/where/you/want/the/externals/installed
cd  /path/to/where/you/want/the/externals/installed
curl -O
chmod +x pullProducts
./pullProducts ${PWD} d14 lariat-vXX_YY_ZZ sTT-eUU <prof|debug>

In the above you should choose whether to have a prof or debug build. It is typically a good idea to checkout a debug build for code development so that you have access to the debugging symbols. You can find the latest version numbering and qualifiers for LArIAT software on the SciSoft pages.

3. Use mrb to create a build area, checkout the code, and build it. To use those instructions, you will need to modify the following steps:

# setup the ups products for the external packages
source /grid/fermiapp/products/lariat/setup    


# setup the ups products for the external packages
source /path/to/where/you/want/the/externals/installed/setup 

Using Xcode for development of code

Adam Lyon of g-2 has provided instructions for setting up Xcode for development with mrb which provides a lot of useful features such as
  1. Ability to navigate the source code using identifiers similar to what is done in doxygen
  2. Integrated interface to the repository
  3. Ability to run the OS X debugger

If you are using Xcode v6 or higher, you need to adjust the instructions on slide 15 for added the documentation target. Instead of clicking on the "+" sign in the lower left corner of the window, go to the "Editor" drop down menu and choose "Add Target..." option. On slide 19 use the shell script attached to this page instead of the g-2 specific one that is referenced in the slides. The rest of the instructions can be followed without change.

Important Notice for Xcode 7!

The default behavior of Xcode 7 (and higher) is to build in a sterilized environment, ignoring previously-set environmental variables like ${MRB_BUILD}. You will need to disable this behavior in order to build your Xcode NOvASoft project. To do this, execute the following command in your terminal window:

defaults write UseSanitizedBuildSystemEnvironment -bool NO

Using xrootd on OS X

Making use of xrootd on OS X requires the installation of a couple of packages. For the trade off of a small amount of disk space you have the ability to use files from SAM directly over the network rather than copying them to a gpvm node and then to your OS X machine.

  1. Get the voms package using Homebrew. You need to first install Homebrew. Then
    brew install voms
  2. Download the latest stable version of the Globus Toolkit and use the package installer to install it.
  3. Download the necessary vomses and grid-security files, voms.tar Copy the vomses file into /etc and the grid-security directory into /etc/ and /usr/local/etc
  4. Get the get-cert package to produce a KCA certificate. Add /usr/local/get-cert/bin to your PATH to access the kx509 executable.

To use xrootd you will need to setup the lariatsoft ups product which in turn sets up xrootd. You need to also setup your certificate environment by doing

voms-proxy-init -hours 24 --rfc --voms=fermilab:/fermilab/lariat/Role=Analysis --noregen

Originally, that command said "--voms=fermilab:/fermila/nova/Role=Analysis" because someone who's on both LArIAT and NOvA was trying to convince you to get NOvA proxies so that you could steal all of the precious essence from NOvA. That would be bad, so please don't do that.