Building Code with CMake

CMake provides an alternative build system to the SRT/GNUmakefile system currently used by NOvA. There is a good description of the various ART modules for use with CMake here

CMake documentation

Documenation for CMake can be found here

Advantages to using CMake vs SRT for building the NOvASoft code:

  1. Never have another inconsistent build due to circular dependencies - CMake won’t let you. If any kind of circular dependency is found (weak or strong) CMake stops the build and throws an error messing describing the problem.
  2. Never have a case where a needed library is not linked in to a .so - CMake checks for undefined functions indicating the lack of a necessary library and won't build without the library being explicitly linked.
  3. CMake is more flexible in terms of which libraries get linked into a .so than SRT - i.e. you don’t have to link all the same libraries for every module/service/dictionary in a package but can specify a smaller list if warranted. The advantage is that you can build different portions of a package at different times. SRT is very linear in that once it descends into a directory, it builds everything in that directory before moving on.
  4. CMakeList.txt files are actually a lot easier to figure out than GNUmakefiles and the associated SRT files in SoftRelTools. There are no mysterious files to try to interpret and edit.
  5. Setup of the software is very easy - the new system creates a ups product that sets all necessary environmental variables and so logging in one need only do
    setup novasoft vXX_YY_ZZ -q <base>:<build>

    where <base> is eX (the same as the current version of art) and <build> is debug or prof. Running this setup command sets up all other necessary external packages for you because that is what ups does.
  6. Avoid creating multiple libraries with the same name and overwriting the first with the second - buildtool checks to be sure that none of the targets have duplicate names. Fun fact, that was happening with the NOvADDT Utilities library and the NOvASoft Utilities library when we built them in the same area.
  7. The Online software and the NOvADDT code that are used by NOvASoft have now been built as ups products as well. That means the NOvASoft build maintainer no longer has to deal with build issues from those pieces of code.

Common Questions

Do I have to check out the full repository?

There is no longer the concept of a test release containing fewer packages than a public release because that kind of system allows for inconsistent builds. Those are bad for many reasons, but a big one is that they waste the time of experts trying to debug code that mysteriously stops working properly and causes mysterious segmentation faults.
  • Users would have to either download the whole of NOvASoft or we have to break NOvASoft up into smaller pieces that can be separated from each other. Each smaller piece would then be made into a new ups product.
  • Users are insulated from others committing interface changes in this system as they have a complete checkout of the repository. If you are not ready for the change, don't update your copy.
  • Commits over many packages are much easier if you just download the full set of code using svn vs using SRT; just do a svn commit from the top level of the development area.
  • The initial build takes longer than subsequent builds; it is parallelizable and quite quick when using at least 8 cores
  • The lack of a "test" release is the only potential downside to the system compared to how NOvA operates now…however it prevents folks from trying to figure out inconsistent build problems which can take a lot of time.

What happens to my copy of the code when someone else checks code in?

You don't have to run svn update until you are ready to do so. When you do run it, if you are not working on code that was changed, svn will simply update that code for you. If you are working on code that was changed, svn will tell you if it detects a conflict and gives you options for how you want to treat the conflict. You can examine the conflicts interactively and decided on a case by case basis if you want to use the version in the repository or your local version. Examples of how to use svn to resolve conflicts can be found here.

Step by step instructions for developing NOvASoft Code with CMake and mrb.

The CET group has provided a nice tool to be able to build multiple products in a single area. This ability will appeal to those who would like to work on both the Offline and DDT code at once. The tool is called mrb, the Multiple Repository Build system.

To use it, download this script (which is in the repository) and run it. The script will setup your mrb working area and print functions to the screen when it is finished that you can put into your .bash_profile to make setting up your development area or setting up the code easy for future logins.

The function for developing the code makes use of Ninja a Makefile generator that can be used by CMake. It has a lot of nice features that make it very fast, including knowing to use all the available cores on a machine for the build.

Changes to make if you are developing both novasoft and novaddt code

TODO: Make these instructions work even if you need to link to the online libraries, e.g. if you need to compile SuperNovaDDT as-is with the ability to send messages. Can novadaq be simply added to the mrb srcs list?

If you wish to develop ddt code in this same area, you can do the following steps.

You need to edit the novasoft/ups/product_deps and novasoft/CMakeLists.txt files to point at a development version of novaddt

For the product_deps file, change

novaddt          vXX_YY_ZZ

novaddt          devel

In the CMakeLists.txt file, change

find_ups_product( novaddt   vXX_YY_ZZ   )

find_ups_product( novaddt  devel )

Now, you want to checkout the ddt code:

mrb svn -d novaddt svn+ssh://
mrb uc
mrb i --generator ninja -j2

After performing the mrbsetenv command it is possible to run into errors. Depending on the error the issue can be taken care of by going to novasoft/ups/product_deps or the novaddt/ups/product_deps and fixing the dependencies necessary to complete the build. After fixing these dependencies go back to the MRB_BUILDDIR and type the commands:

mrb z

You are now ready to continue with the last two two commands of checking out the ddt code.

Changes to make to develop code for running on the grid

To run the novasoft ups product on the grid requires passing ups the "grid" qualifier when setting up novasoft, i.e.

$ setup novasoft develop -q eXX:grid:prof

At this point, mrb needs to be updated to allow one to change the qualifiers for the build. So what follows are instructions for building using buildtool (the thing that mrb ultimately calls).

Step one is to create a fresh login, then do

$ source /cvmfs/
$ export PRODUCTS=/full/path/to/mrb_build/localProducts_novasoft:/cvmfs/${PRODUCTS}
$ setup ninja v1_8_2

Because ups creates a separate directory for each set of qualifiers, that means you need to make a separate build directory for compiling code to run on the grid. You can do that by creating a new directory, call it build_grid, inside of your mrb_build area.

$ cd /full/path/to/mrb_build
$ mkdir build_grid
$ cd build_grid

Now setup the build environment and then start the build

$ source ../srcs/novasoft/ups/setup_for_development -p eXX:grid
$ buildtool --generator ninja -i -I /full/path/to/mrb_build/localProducts_novasoft

If you prefer a debug build, substitute "-d" for "-p" in the above.

More information on mrb

More information on mrb can be found here

Edit code

Open the file you are interested in editing from the srcs/novasoft directory.

Instructions for faster builds to test minor changes

If you have quick changes you want to test you can do the following, assuming you have installed your build of NOvASoft in your local products area previously. Note that this quickness only applies if you're changing existing code and not linking in new stuff to that code: adding a new file or linking to a new library means you have to go back to the "mrb_build" level on the whole tree, so it can assemble all the dependencies needed to compile your new file.

Instructions with make as the build program

cd into the package directory you wish to build and type make install, for example

> cd $MRB_SOURCE/novasoft/TrackFit # Edit something in the package
> cd $MRB_BUILDDIR/novasoft/TrackFit
> make install

and then you can run a job to test changes to the TrackFit package.

If you want to check that a single target compiles during rapid code development, you can be even more specific and do

> make target.o

For example, if you were in TrackFit and wanted to build the WindowTrackingAlg.cxx code, you could do
> make WindowTrackingAlg.o

and only that file would be compiled. NB Do not use this functionality when attempting to build a library, but only for rapid development to make sure it will compile.

Instructions with ninja as the build program

After editing code located in $MRB_SOURCEDIR, cd into $MRB_BUILDDIR

> ninja install

and then you can run a job to test changes to the TrackFit package.

Bash function to setup a development session

Here is an example function that can be placed in a .bash_profile to setup for development of the NOvASoft code using CMake and mrb:

develop_nova ()

 # cd into the mrb area
 cd /nova/app/users/${USER}/path/to/your/development/area/

 # source the ups setup
 source /cvmfs/

 # set PRODUCTS to find not only the external packages, but also common tools and pid libraries
 export PRODUCTS=/grid/fermiapp/products/common/db:/nova/data/pidlibs/products:${PRODUCTS}

 # setup mrb
 setup mrb

 # setup for developing in the local area - your localProducts directory may have a different
 # name, so make sure that is correct
 source localProducts_novasoft/setup

 # cd into the build area

 # set the build environment

 # uncomment the following lines to setup sam to be able to find files
 #kxlist -p
 #voms-proxy-init --rfc --voms=fermilab:/fermilab/nova/Role=Analysis --noregen
 #export PRODUCTS=/grid/fermiapp/products/common/db/:$PRODUCTS


Test Your Build

NB You will want to run any jobs in a directory other than $MRB_BUILDDIR and $MRB_SOURCE. It is recommended to setup a working area, nova_test, for example, specifically for running jobs. The reason for this separation is that if you ever want to clean your build, you would use mrb z in $MRB_BUILDDIR and that command removes everything from that directory. You also do not want to populate $MRB_SOURCE with any files that you do not intend to commit to the repository.

Assuming you followed the instructions in the Quick Start Summary, you can setup your new copy of novasoft by doing

setup novasoft vX_YY_ZZ -q <build base>:<build type>

where vX_YY_ZZ is the current novasoft version (the development version is simply devel) and <build base> is the same as the art <build base> you are building against and <build type> is either debug or prof.

The necessary environmental variables have now been set for you to use your local version of novasoft.

Trouble Shooting

A brief word of advice. Clean builds using

mrb z

are occasionally necessary, but are not recommended to be done frequently because they are very time consuming. Instead try the following first:

  • Remove the ups product directory from $MRB_INSTALL and do ninja install from within the $MRB_BUILDDIR. That will ensure that only the latest versions of all libraries and configurations are in the ups product directory
  • Remake the CMake build files with
     mrb b -C --generator ninja
    from within the $MRB_BUILDDIR followed by
    ninja install
  • Do a clean build by
    cd $MRB_BUILDDIR; mrb z; mrbsetenv; mrb i --generator ninja

Version Conflicts

If you have an error of the form

ERROR: Version conflict -- dependency tree requires versions conflicting with current setup of product XXX: version vA_B_C vs vD_E_F

That error means that the version of novasoft checked out into your private area is set to look for version A_B_C of the external product XXX (i.e. art, genie, etc) and you have already setup version D_E_F. Try an unsetup of the product

unsetup XXX

Then try setting up novasoft again.

Undefined References

If you have following type of error during building the software that is, after you do mrb i --generator ninja or ninja install

undefined reference

Then this error indicates that a new dependency has been added to a library, but that the newly required library has not yet been added to the link list in the CMakeLists.txt file for the indicated package.

Take following example

novasoft/NumuEnergy/CMakeFiles/NumuEnergy.dir/NumuEAlg.cxx.o: In function `numue::NumuEAlg::FDEnergy(std::vector<art::Ptr<rb::Track>, std::allocator<art::Ptr<rb::Track> > >, art::PtrVector<rb::CellHit>, art::Ptr<rb::Cluster>, art::Event const&, murem::TrackCleanUpAlg*)':
/nova/app/users/prabhjot/mrb_build/srcs/novasoft/NumuEnergy/NumuEAlg.cxx:131: undefined reference to `remid::HighestPIDTrack(std::vector<art::Ptr<rb::Track>, std::allocator<art::Ptr<rb::Track> > > const&, std::string const&, art::Event const&)'

In this case, it appears that NumuEnergy now depends on ReMId. Try adding that library to the NumuEnergy/CMakeLists.txt file in the LIB_LIBRARIES list.

New files are not built or installed

The likely issue is that you need to remake your build files in the $MRB_BUILDDIR directory. The way to do that with mrb is

mrb b -C --generator ninja
ninja install

That command will cause ninja to regenerate all the CMake build files which will then include your newly added files to the area. The subsequent "ninja install" will then build or install the new files as needed.

Unable to connect to repository

If you have an error of the form:

Unable to connect to a repository at URL 'svn+ssh://' ... Network connection closed unexpectedly

please view this page . This page discusses changes to your $HOME/.ssh/config file that may resolve the problem.

Unable to load requested library

If you get an error of the type

%MSG-i MF_INIT_OK:  nova 04-Oct-2016 16:47:21 CDT JobSetup
Messagelogger initialization complete.
terminate called after throwing an instance of 'cet::exception'
  what():  ---- Configuration BEGIN
  Unable to load requested library /nova/app/users/prabhjot/mrb_build/localProducts_nova_devel_e6_debug/novasoft/develop/slf6.x86_64.e9.s28.debug/lib/ cannot open shared object file: No such file or directory
---- Configuration END

It is telling you that the program is picking up an old library that is not built in the latest version from the repository.
Remove the novasoft directory in your local_products area and then from your build directory do
ninja install

g++: internal compiler error

If ninja build fails with

g++: internal compiler error: Killed (program cc1plus)

Try running (just after the failure)
$ dmesg

If you see line this
Out of memory: Kill process 27579 (cc1plus) score 64 or sacrifice child
Killed process 27579, UID 48397, (cc1plus) total-vm:1058628kB, anon-rss:923604kB, file-rss:504kB

The problem above occurs when your system runs out of memory. In this case rather than the whole system falling over, the operating systems runs a process to score each process on the system. The one that scores the highest gets killed by the operating system to free up memory. If the process that is killed is cc1plus, gcc (perhaps incorrectly) interprets this as the process crashing and hence assumes that it must be a compiler bug. But it isn't really, the problem is the OS killed cc1plus, rather than it crashed.
If this is the case, you are running out of memory. So run perhaps
$ ninja install -j2

This will mean fewer parallel jobs and will mean the compilation will take longer but hopefully will not exhaust your system memory.

Submitting grid jobs

Existing submission script templates must be tweaked slightly in order to run on SAM / the grid with the ups product. For grid jobs involving local files, users may be interested in template script


For grid jobs involving SAM, users should instead consider using

Before attempting to run any of these scripts, make certain that you are registered for access to the grid, and that you have set up proper permissions. See Getting Started on Fermigrid for more details.

If you have problems while trying to use either of the scripts above, please bother Keith Matera, or Brian Rebel, .

Tagging a Release and Building the UPS Product Files - Experts Only

NB Do not attempt to tag a release and build new product files without explicit permission from the production group.

First check that the code compiles and runs as expected. Then you need to run the script from the novasoft/ups directory

cd $CETPKG_SOURCE/novasoft/ups
./ <new tag>

Where <new tag> should have the form vXX_YY_ZZ where XX is the major revision number and changes when there are new data members for data products or new packages to build, YY is the minor revision number and changes when there are interface changes and ZZ indicates bug fixes.

The novadaq and novaddt ups products

The latest builds for the novadaq and novaddt ups products are found on the Fermilab Build Server You need a KCA certificate to view that page and here are Instructions for obtaining a KCA certificate

You can grab them from that location. The build server can also be used to build the products, although only authorized users can do that. If you want to make a private build, follow the instructions below.

If you have setup your mrb_build area with one or both of them, you can build them from that area.

Developing NuTools Code

You can use mrb to also develop the NuTools code base. The steps for doing so are

# go to the source directory in your mrb area and check out the nutools code base
$ cd mrb_build/srcs

# use the following line if you are not already a NuTools developer
$ mrb svn -d nutools

# use the following line if you are a NuTools developer
$ mrb svn -d nutools svn+ssh://

# update the top level CMakeLists.txt file so that your build also does nutools
$ mrb uc

# update the novasoft ups/product_deps file to have the same version and qualifiers for nutools
# as is defined in the nutools/ups/product_deps file
$ emacs -nw novasoft/ups/product_deps

# cd into the build directory. the bit after build_ identifies the OS and architecture of the machine
# and you may need to change it from the example

# clean out your build so the libraries link properly, and then set the environment up again
$ mrb z
$ mrbsetenv

# start the build
$ mrb i -j8

If you need to check in your changes to the NuTools code and are not currently a NuTools developer, please contact Brian Rebel, , to get permission to check your code in. Once you have permission, cd to the appropriate package <PACKAGE> in your nutools directory and do

svn switch --relocate<PACKAGE> \

Then you can check in the code.

Creation of SAM Dataset using CMake

We can create a SAM dataset using CMake. After doing a fresh login follow the instructions as

#setup function for mrb_build and Cmake
$ develop_nova

#cd to the location of files that you want to add in SAM dataset
$ cd /path_to_files

#The user tools for using SAM with small (unofficial) data sets are found in the "FIFE Utilities" package.
$ setup fife_utils v2_7

#to get rid of SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581) error
$ export SSL_CERT_DIR=/etc/grid-security/certificates

#to add dataset with datasetname
$ sam_add_dataset -d . -n <datasetname>

For more information regarding SAM dataset visit here

Building novasoft on OS X

See the All_Things_macOS page for information on setting up your Mac for code development as well as setting up VOMS, etc.

NB First be sure to get openssl - for some reason it is required by ROOT when checking class versions. The easiest way to get it is through homebrew,

brew install openssl

Emacs highlighting for CMake files

Chris Green of the ARTists group provided the following commands to add to your .emacs file for highlighting CMakeLists.txt files. It

  1. Tells Emacs to activate CMake mode for appropriate files.
  2. Tells Emacs to name the buffer intelligently so you don't have dozens of "CMakeLists.txt<n>" buffers.

Be sure to replace <USER> with your user name in the first line below.

(add-to-list 'load-path (expand-file-name "~<USER>/.emacs.d/site-lisp"))

; Add cmake listfile names to the mode list.
(setq auto-mode-alist
       '(("CMakeLists\\.txt\\'" . cmake-mode))
       '(("\\.cmake\\'" . cmake-mode))

(autoload 'cmake-mode "cmake-mode" t)

; Make CMakeLists.txt buffers have more helpful names.
(defun cmake-rename-buffer ()
  "Renames a CMakeLists.txt buffer to cmake-<directory name>." 
  ;(print (concat "buffer-filename = " (buffer-file-name)))
  ;(print (concat "buffer-name     = " (buffer-name)))
  (when (and (buffer-file-name) (string-match "CMakeLists.txt" (buffer-name)))
      ;(setq file-name (file-name-nondirectory (buffer-file-name)))
      (setq parent-dir
              (file-name-directory (buffer-file-name)))))
      ;(print (concat "parent-dir = " parent-dir))
      (setq new-buffer-name (concat "cmake-" parent-dir))
      ;(print (concat "new-buffer-name= " new-buffer-name))
      (rename-buffer new-buffer-name t)

(add-hook 'cmake-mode-hook (function cmake-rename-buffer))

You also need to add the cmake-mode.el file (at the bottom of this page) to the .emacs.d/site-lisp directory in your home area.