Documentation FOR BEGINNERS » History » Version 160

« Previous - Version 160/167 (diff) - Next » - Current version
Susan Lein, 04/22/2016 08:46 AM

Documentation FOR BEGINNERS

Hello! This page is especially geared for those very new to the NOvA software. Feel free to edit this page yourself.

To begin, you need to make a test release of the NOvA code. This allows you to work with the code and make changes to it. Don't worry about your changes hurting other people -- they are only added back into the NOvA code when you "svn commit," and you won't do that until you understand the consequences.

Log into Fermilab machines

First, log into the fermilab machines. Two pages with further info about this are Fermilab Computing Access and Using NOvASoft on the GPVM nodes. You need to get accounts set up with Fermilab -- this means do ALL the steps listed on Fermilab Computing Access, all the way down to getting yourself on the email list and getting access to the svn repository. Do not proceed until all have been accomplished.

To check to see if you have been added as a developer, see if your name is listed on the Overview Page.

Now, log into the novagpvm machines. Take your pick -- the list of ones to choose from is at the top of Using NOvASoft on the GPVM nodes

Then, you need to change to your working directory.

% cd /nova/app/users/yourname

(Here and elsewhere, the '%' is a Linux terminal prompt). You should pretty much forget about the place you go to when you first log in -- do all of your work in /nova/app/users/yourname. For instance, I always do
% cd /nova/app/users/lein/

before I begin anything. If you don't already have a directory here, file a service ticket (Service Now, logging in with your Services credentials) to get one. The directory should be named based on your principal to keep the top directory neat and tidy.

Working with NOvA Code

Next, you need to tell the computer about the NOvA code. There are two strains of the novasoft that are in use. The most important is 'development'. This is the most up-to-date codebase and, as the name suggests, is in perpetual development -- it is forever evolving! The second option is a 'tagged release'. The offline group periodically make a tag of the development and process Monte Carlo and Data files using this tagged release. In essence, the tag is a snapshot of development when it was in a known, stable state. This is done in order to have everybody using the same processed Data/MC files that are produced with a stable novasoft.

You are encouraged to add the following lines to your ~/.bashrc or ~/.profile files, which are located in your machine once you are logged in, so that you do not need to remember the above incantations:

function setup_nova {
  source /grid/fermiapp/nova/novaart/novasvn/setup/ "$@" 
  cd /nova/app/users/YOUR_USER_NAME_HERE

Once you have this then, from now on when you log on to a nova node you can simply type:

% setup_nova (sets up development release)

% setup_nova -r <tag>  (sets up a tagged release)

Making Your Own Test Release

Now you can make your new release. You can see Editing Code for full information on this. If you just wanted to get started now with development code,

% newrel -t development dir_name  

or if you need to base your test release on a tagged release, in particular the tag S12-12-12 version, do
% newrel -t S12-12-12 dir_name  

Please change dir_name to whatever you would like it to be, for instance, test_novasoft. Then

% cd dir_name

Once there, since you have already sourced the novasoft code (either development or a tagged release), you should setup your local code.

% srt_setup -a

The 'srt_setup -a' is what tells the build system to use your local test release first and hence to pick up any changes you may be making to the code. I find as a beginner it is good to be paranoid about this command and do it compulsively. If your code isn't working and you don't know why check if the code knows it should be giving your code priority over the development code by checking the following:

The first option tells you what release you originally source, the 'public' development release will show to your terminal window

and likewise if you have originally sourced a tagged release it will show

where '<tag-name>' is the label for the tagged release.

The second option tells should tell you the location of the test release you're working with:


Adding Packages to Your Test Release

If you want to alter the code at all, you need to check out packages. Full information again exists at Editing Code, but basically you need to

% addpkg_svn -h package_name

in case you are working with development release.
For a tagged release, follow this example:
% addpkg_svn package_name <svn-tag> 

where <svn-tag> is like this S12-12-12 for example.

*N.B. The package information page needs lots of work
The very incomplete and out-dated list of packages and very limited information about them can be found here: Complete list of NOvASoft packages.

Then, once you have made your changes you need to "build" your software. See section below on Building Software and Clean Builds.

NuTools Packages

This is really not a beginners topic, but maybe you've wondered what all these "Base" things are about (eg. EventGeneratorBase, EventDisplayBase etc.) This is code shared among IF experiments at and any development and changes to the code are coordinated by the NuSoft-ART project. You can ask for developer status from the project manager, currently Brian Rebel (), to improve these tools and add new functionality.

NOvA has chosen to include the nutools packages as a ups product, and the development of those packages has changed. Just including those packages in your test release and building per normal will result in inconsistent builds and unexpected behavior. Please check the NuSoft-ART wiki for the most up to date information: Developing NuTools Code

Removing Packages from Test Release

Currently, you need to add the -f option to remove a package without being annoyed:

% rmpkg -f <pkg> 

This will do a "make.clean", then remove the symbolic link from the test release include directory, and then remove the directory. This is why you should NOT simply do:

% rm -rf <pkg>

NOvA Software File Naming Conventions and Package Structure

In our software there are modules, products. A package may have one or more of these types, although preferably only has one.

Modules are the objects you will most likely be modifying and developing. These pieces of code let you put products in events, and analyze products already in an event. They come in three flavors: Analyzers, Filters, and Producers and all carry module in their name (eg. The package will have two configuration (.fcl) files, one to define parameter sets for the module (eg. CalHit.fcl) and the other as a sample job configuration (how the module is to be run, eg. calhitjob.fcl). For more about modules see Chris' tutorial from the April 2014 collaboration meeting, The_NOvA_Offline_Workbook

Products are objects to be put into files and used for analysis. They will have a header file (.h) with class definitions and everything that goes with that class. They will also have an implementation file (.cxx) which will have the function definitions. In the package there will also be a classes.h and a classes_def.xml file which handle dictionary generation and class versioning. A good example for this structure is Geometry Objects. For more on commonly used products, see Evan's tutorial from the April 2014 collaboration meeting, The_NOvA_Offline_Workbook.

Do not commit compiled code to the repository (*, * These are produced by the ART/ROOT framework compilation and having a permanent copy of them will confuse the compiler and svn.

There are also ART services that you might run across (eg., for their use checkout Chris' tutorial and this extensive documentation ART services from the Mu2e collaboration. This is not a beginner's topic so you might find yourself just copying pieces of code, the tutorials in the Demo package are a good start, but for those interested in upping their ART fu, happy reading!

Module Labels

Modules have names. These are DIFFERENT from their labels. A module label is what, for instance, a job.fcl file uses to call that module. It is like a variable name -- the label can be anything. You might want to call the same module five times in the same job -- then you would need five different labels. Thus, the label could be Fred. Or Susie. Anything. However, we do try to make sensible labels so humans easily understand what is going on.

This label, then, might be called later in some other file. For instance, in the DataCheck module, we need the label of the Slicer module because we want to use the clusters created by Slicer. NOTE -- we are using the label, not the name. Thus, it could change depending on what job.fcl file we made to run the DataCheck and Slicer modules. We do not want to hard-code module labels then, because it will be very difficult to debug later when we forget and change the label. Labels should be passed to the module in the configuration .fcl file as parameters. See section on how to Adding a Parameter for information on how to do this.

If you need to find the label of a module .fcl job file that runs over, look at the end of the .fcl file. You will find lines like:

physics.producers.calhit:        @local::standard_calhit
physics.producers.slicer:        @local::standard_slicer

Then you know that the module label for CalHit is "calhit" and the module label for Slicer is "slicer." Whatever is listed after physics.producers.**LABEL OF MODULE**** is the label of the module. (This of course holds also when you have an analyzer instead of a producer).

List of Libraries To Link

In one's package's GNUmakefile, one can link to libraries. You might be adding to your code and find that you get error messages like:

nova: symbol lookup error:
undefined symbol: _ZN2rb7ClusterD1Ev

This might mean you need to link another library, check the Trouble Shooting and Gotchas. These are likely libraries of novasoft code. For instance, in that case, I had wanted cluster information but the previous code had never needed it. So, one needs to determine WHICH library to add. To find a list of libraries,


Once you find the library you want to add, put it in the list of other libraries inside the particular package's GNUmakefile file. In my case, I needed to add libRecoBase (which knows about clusters). These additions are added onto the line that might look like:

override LIBLIBS += $(LOADLIBES) -L$(ART_LIB) -lart_Framework_Services_Optional_TFileService_service -L$(SRT_PRIVATE_CONTEXT)/lib/$(SRT_SUBDIR) -L$(SRT_PUBLIC_CONTEXT)/lib/$(SRT_SUBDIR) -lChannelInfo -lCMap -lTrackFit -lRecoBase -lRawData -lGeometry

One can see that I have added libRecoBase, specifically it is the -lRecoBase part. Add your new library to the end in the form of -lLibraryName.

Remember to clean build after things like this.

Output Messages

Currently, there are two ways nova software deals with output statements. In one, messages get sent to output files like cerr.log and warnings.log. This is the current default. If, however, you would like your output statements to go to the screen (from where you can put them into another file if desired) you need to make the following changes. In your runjob.fcl, you need to add

#include "job/messageservice.fcl" 

at the top with your other include statements. Then, with your other services statements, put
services.message:          @local::standard_out

Then everything will get output to the screen. If you want this to be saved into a file instead, simply follow your regular run job command with >logfilename.log, i.e.

% nova -c job/runcalhit.fcl -s inputfile.root > inputfile.log

Running over more than one file

You can run your job over more than one file. Instead of doing -s filename.root to your job command, instead do:

% nova -c job/<.fcl file> -S <textfile>

The text file should contain a list of all the file names you want to run over. To get a text file that contains the name of all of a type of file, for example, do:

% ls -d -1 /nova/data/mc/S11.04.30/genie/ndos/sim_genie_nhc_simpleflux_swapping_none* >  textfile.txt

This will return a file with all of the sim_genie_nhc_simpleflux_swapping_none* file names.

Coding Preferences

PLEASE code clearly and neatly and follow our coding conventions and other tips from Editing Code! Please document well what your code is doing and why. Also, please try to code neatly. Human-readable code is far more useful if things line up neatly in columns - it makes it much easier to debug and find problems. For an example of nice, neat, column-wise code, consider this example from Cana.ccx (in the Commissioning package):

  fNhitTotal   = f->make<TH1F>("fNhitTotal",  
                   250, 0.0, 2500.0);

  fNSlices     = f->make<TH1F>("fNSlices",

  fNhitInTime  = f->make<TH1F>("fNhitInTime", 
                   200, 0.0, 200.0);

  fNhitOutTime = f->make<TH1F>("fNhitOutTime",

  fNtimeWin    = f->make<TH1F>("fNtimeWin",
                   ";N time slices;Events",
                   10, 0.0, 10.0);

  fNhitTimeWin = f->make<TH1F>("fNhitTimeWin",
                   ";Nhit;Time slices",
                   200, 0.0, 200.0);

From later in that same file,

  fTimeWindow   = p.get<float>("TimeWindow");
  fNhit         = p.get<int>("Nhit");
  fNhitX        = p.get<int>("NhitX");
  fNhitY        = p.get<int>("NhitY");
  fSlicerModule = p.get<std::string>("SlicerModule");

Or from Cana.fcl, we again see neat organization of code:

TimeWindow:     0.5             # Size of search window in usec
Nhit:           12              # Total number of in-time hits required
NhitX:          4               # Number of hits required in x view
NhitY:          4               # Number of hits required in y view

One further courtesy: We are collectively responsible for the analysis code and there may be times when you will be editing code that was not originally written by you. In such cases, look at the layout style the author used and try to make your code fit in. If you find code that you find confusing and could benefit from some tidying, go ahead and tidy things up for the next person who might look at the code.

A Few Coding Hints

NOvA software will have you often NOT working in the standard namespace. Thus, one should always do std:: cout, std::endl, etc. instead of cout, endl, etc. When one works with strings, you need to (at the top of your .h file or wherever you are using it) add #include <string> and then, when creating a variable of type string, remember to do:

    std::string  VariableName;

Building Software and Clean Builds

Building software is how the computer compiles together all the nova software into a usable form.

Sometimes you need to make a "clean build" because the base code changed enough or the ART framework was updated or it is the first time you have built the software. You can NOT simply "gmake clean" and then "gmake" as you could in FMWK (previous nova code). Now, the NOvASoft packages need to built in a specific order (which gmake doesn't know how to do). There is a script that handles a clean build for you. The only thing that you need to do before running the script is make sure that your $SRT_PRIVATE_CONTEXT has been defined to point to your test area. You did this earlier when you

% srt_setup -a

Once you do that, run the script:
% $SRT_PUBLIC_CONTEXT/SRT_NOVA/scripts/novasoft_build -t

at the top of your test release. This is the same as doing the old version of:
% gmake clean  (gmake clean means that next time you gmake it will start from scratch again.)
% gmake

If you don't do this you will get error messages that allude to the fact that libraries haven't linked.

Later, you may make small changes to small parts of the code in your test release. You don't need to do a brand new build every time. If, for instance, I have changed a few files in CalHit, this is all I need to build. Then I would just do (at the top of my release)

% gmake CalHit.all

This is much faster than a whole build. If you are working with the Demo package or if you have used the Demo package to make your own different package, you need to individually build these ALWAYS. The build script doesn't know about these packages so it doesn't take care of them. So you would need to do:

% gmake Demo.all

Or, if you wanted a clean build of the Demo package (or some other package you have made) then do:

% gmake Demo.clean
% gmake Demo.all



For instructions on accessing and using the devs svn repository, check here Devs SVN Access.

SVN Update

You are encouraged to check how your codes compares with the repository:

% svn status -u package_name

People add new code into the development release all the time. If you have a package checked out in a release, it will not automatically get updated to match these updates. That is a good thing, because you don't want your code breaking all the time because others are changing things. But sometimes you do want these changes. Then, to update an entire package, from the top of your test release directory, do:

% svn update package_name

To update just one file, do:
% svn update package_name/file.something

Also, to update all packages in your release:
% svn update *

When svn is updating, you will get output of file names with a letter code. You should look at this and think about it -- does it match what you expected? The letter codes are:

  • G is Merge -- svn is being clever and merging the differences between your code and the code in the repository. This is fine.
  • P is Patch -- there was only a small change which is being sent as a patch. This is fine.
  • U is Update -- there was a larger change in the base code and it is being updated. This is fine.
  • C is Conflict -- there is a conflict between your code and the base code. THIS IS NOT FINE. Go to the listed files for conflict. Look at them; conflicts will be flagged with '>>>' and '<<<'s. Edit to make the code make sense for your purpose.

Please remember to gmake after you have done things like this!

SVN Commit

This is what you do when you have finally reached the point of creating code others want to be able to use -- you are going to put your code updates back INTO the development release. In the beginning, do NOT do this lightly. You could break everyone's code if you commit things that do not work. HOWEVER, once you are at the point where you know what you are doing, please DO SVN Commit. The mindset should be "early and often" -- this is a way to backup your code. If you had done a lot of work and DIDN'T svn commit and your code is lost or deleted -- then it is gone forever. So don't wait until everything is in a perfect final state, but on the other hand, be mindful in the beginning to NOT break everybody's code.

OK - on to the procedure. If you think that the development code in the package has been updated since you last checked it out, again you can check this by svn status -u, you might want to do a svn diff FIRST to pickup and resolve any of those differences:

% svn diff -r HEAD package_name

This will list the differences between the base version and yours. Look through the differences and make sure they are all things you are purposefully changing -- often stray cout statements for debugging, etc. try to sneak through. Watch for these.

Lastly, once you are sure you should be doing this (again at the top of the package you want to update):

% svn commit

This will pop open an editor (see next section for more about this). Below the lines already given, please write in your notes about this update. Save and exit the editor. You are now done! You should soon receive an email from SVN Update about your change!
Alternatively, you could do:
% svn commit -m "Your message here that you are not going to type into an editor" 

This way, you don't get any editor at all and your message is just passed along.

To add new files to the repository, you need to follow the same general procedure but instead of just doing svn commit, first do:

% svn add filename

Then next do
% svn commit filename

Now document, etc. as above.

If you want to commit multiple files at once from the same package but NOT the whole package and all the files should have the same message, you can do

% svn commit file1 file2 ... filen

with an editor OR
% svn commit -m "Your message" file1 file2 ... filen

In case you have mistakenly done svn add for files that you did not want to commit, but not yet committed, you can do the following to take them off the list of files that svn is determined to commit.

svn rm --keep-local file1 file2

So next when you do svn commit, all files that you svn added will be committed, other than file1 and file2.

Whenever you commit, include an intelligent commit message. These messages go out to the novasvncommit mailing list and are what people read to find out how the code has changed. It's a good idea to sign up to receive messages from this list to keep abreast of changes to packages you're working on.

SVN Commit E-Mails

The notifications are sent to the NOVASVNCOMMIT listserv, so simply subscribe by sending an email to with the following content (substitute "First" and "Last" with your first and last name of course):


SVN Commit Editor (and .bashrc files)

FIRST -- see PREVIOUS section on what to actually write in your svn commit editor and where. This is only further info about making the editor the one you want, etc.

When you svn commit, an editor pops up. This might be VI, it might be something else. If you are like me and are addicted to emacs, you can do the following. You can execute the following command BEFORE you do svn commit.

bash% export EDITOR=emacs

If you want to make this permanent, add it to your .bashrc file. OK -- so do you know what a .bashrc file is? If not, here is the scoop. This is a super helpful file that you setup for your account. I like to put things in it like alias' so I don't have to type out the whole command for setting up scripts, etc. Also, for some reason, my fermilab account actually reads from my .bash_profile file instead of my .bashrc file.

I think it is best if you just make these the same -- then you probably won't get screwed up. If you change your .bashrc / .bash_profile files, source the file to see the updated behavior (source ~/.bash_profile).

If you are curious about the difference between the two files and how to set them up to be in sync, check out Bash

These files need to live in your ~ directory -- not your /nova/app/users location.
This means, to get to the location,

 cd ~

Then to see if you have it,
ls -a
(in your ~ directory).
To edit it using emacs
% emacs .bash_profile &    (or .bashrc)

Just in case it is useful to you, this is my .bash_profile file. The first line is necessary for all .bashrc/.bash_profile files. The next line is just because I like how it formats my command line. Then I make my editor emacs by default on the next line. I made alias' for the source code next, followed by an alias for using the script to clean build. This means that I just type "snova" and it knows that I am really meaning that long line of code to source. Lastly, I alias rm, cp, and mv so it makes sure I really want to do that before I copy over something, for instance.

export PS1='[\!]\u@\h \w> '
export EDITOR=emacs
alias novagmake='/grid/fermiapp/nova/novaart/novasvn/releases/development/SRT_NOVA/scripts/novasoft_build -t'
alias rm='rm -i'
alias cp='cp -i'
alias mv='mv -i'

.fcl File Hints

There are two types of files ending in .fcl. Some of these files are configuration files, files that define different configurations of which to run your module. For example, in the CalHit package, there are two .fcl files. CalHit.fcl is the configuration file. It defines two ways of running, one for mc files and one for data files. Configuration .fcl files will always be named PackageName.fcl. Then there are "run job" .fcl files. These are the files that you would actually execute with the nova -c job/file nomenclature. For CalHit, this is the calhitjob.fcl file. Job .fcl files will always be named packagenamejob.fcl. If you understand these two files, it should be relatively easy for you to look at other .fcl files and decide if they are configuration or run files.

Every run job .fcl file has the following line:

 maxEvents:   -1 (or 10 or 100 or some other number) 

This tells the program how many events to run over in your file. If you set it to 10, you will miss most of the file. But sometimes you just want to do a really quick test, in which case, 10 events is just fine. In most cases, you will want to run over the whole file. Don't try to just set the number to some astronomical number. Our data files are really big, so you might miss some anyways. Instead, set the number to -1. Obviously, counting up by one starting from one, you can never reach -1 and so will run over the whole file. One might be tempted to omit the line altogether to run over the whole file. THIS DOESN'T WORK!! Having said that, it DOES work if you add
'-n -1' at the command line, for something like:
% nova -c job/file -n -1....

You don't want to have to remember this, so just leave the line in. You can always override it at the command line as above. For instance, when you want to do that quick test, just add -n 10 at the command line instead of having to change your code.

NOvA Data

More information about the NOvA data files can be found at Data.

Raw Data Naming Conventions

Raw data files are named as follows:


  • det is a detector name (ndos,nd or fd)
  • rxxxxxxxx is a unique run number (8 digits)
  • sxx is a subrun number (2 digits)
  • stream is eighter mask for a trigger (calibration, numi, cosmic,etc...) or trigger number in a txx format (t followed by two digits).

The stream names correspond to:

  • There is no stream name for files with a global trigger only -- everything from that run is written into this file.
  • t00 is the NuMI trigger.
  • t01 is the Booster trigger.
  • t02 is the cosmic trigger.
  • t04 is the calibration mode.

What is in a File

Please see the short writeup located at What is in a file to find out how to use a root browser to see what has already been run on a file you are using. This is a good way to check if cal hits has been run already, for example.

Using Event Display

Let's say you want to look at an event in the event display. You have already done:

% nova -c job/evd.fcl -s file.root

Now you want to use the menus to change how you see things. Perhaps you want to see slices. Then go to Edit->RecoDrawingOptions. Since slices are clusters, you first need to change ClusterModules from ["clusters"] to ["slicer"], the name of the module you want to view. If instead, you wanted to see tracks from KalmanTracker, you would change TrackModules from ["tracks"] to ["kalmantrack"]. Now, to see the slices, you need to change DrawClusters as well. You could change the value from 0 to 1, but I prefer to see slices drawing with DrawClusters = 2. Similarly for tracks, you need to set DrawTracks to 1.

Other options exist in Edit->RawDrawingOptions. You can change if the color of the hit corresponds to the hits charge or time by changing ColorShowsCharge and ColorShowsTime. Another popular option is to change ScaleHitsByCharge to 0 - then the size of the hit doesn't correspond to the change but is just the size of the cell.

A few more useful options exist in Edit->SimulationDrawingOptions. You can view FLSHits by changing ShowFLSHits to 1. Setting ShowMCTruthTrajectories to 1 allows you to see the paths taken by the mc particles. Setting ShowMCTruthText equal to 1 displays basic information about the interaction, daughters, and energies of the particles involved.

Searching Code

Imagine that you want to find every instance of, for example, fTriggerMask_TriggerType used in all of nova code. The best way to do this is NOT to use grep. This will take too long. Instead, use

% ack "fTriggerHeader_SourceID" $SRT_PUBLIC_CONTEXT/

If you have sourced yourself so that public context is set up, then this will search all the files in novasoft.

A few words about timing in NOvA

The DAQ uses "nova time" for almost all its timestamps and converts all other time stamps to "nova time". "nova time" is defined to be the number of ticks of a 64 Mhz clock since Jan 1, 2010 (I think...). The point is that there are no leap seconds in "nova time" which other time systems (GPS, UTC, eg.) insert to keep the time sync'ed to the solar cycle.

To convert, a constant is used. This constant kUSEC_PER_TDC is (10^6 usec / sec) / (64 million clock ticks /second) to get usec/clock tick. Later, when I want to convert TDC counts to usec I multiply by this conversion constant.

One should always to this:

dt = t2*kUSEC_PER_TDC - t1*kUSEC_PER_TDC

and not this:

dt = (t2-t1)*kUSEC_PER_TDC;

The reason is that the TDC values are unsigned and so if the value of t1 ever exceeded the value of t2 the subtraction t2-t1 (who's result would also be unsigned) would be a 2's complement of the negative number. Probably not what most people expect. Doing things the first way makes the compiler first convert to double precision (which is signed) and then subtract.

Words of Wisdom

Brian Rebel suggests NOT making a ton of different test releases. This is likely to just going confuse you later, when you want to go back to what you were working on and you can't remember which release you had been working in. Please also make sure you name them decent things. Test23 will not have a lot of meaning to you in one year.

Copy files from Fermi to your machine

Sometimes you want to copy files from your fermi account to other machines -- maybe you made a histogram and need to put it into a talk. To do this, one needs to be logged into the fermi machine. Then simply do:

% scp fermifiles username@othercomputer:/path/to/location/on/other/computer

For me, I would do:
% scp histo_datacheck.root

If you want to copy a directory, add the -r option behind scp, i.e.
% scp -r directory username@othercomputer:/path/to/location/on/other/computer

One other hint: if you need the full path length to a location, do
% pwd

This will then display the path to that location.

Standard NOvA plot style

We have adopted a standard style for blessed plots. See NOvA plot style for instructions on how to configure this as your default.

CRAP! You overwrote / deleted / destroyed your files!!!!!!

It is ok! IF you are working on fermilab machines, in the /nova/app/users area, then snapshots are made of the last four days. To access this, go to /nova/app/.snapshot/. After this, choose the directory corresponding to the day you want to go back to. After that, cd /users and navigate as you would normally.