LBNE-centric (that is, OBSOLETE) Guide to Using LArSoft in git/mrb world¶
Please see The DUNE Getting-Started Guide instead¶
- Table of contents
- LBNE-centric (that is, OBSOLETE) Guide to Using LArSoft in git/mrb world
- Please see The DUNE Getting-Started Guide instead
- Run from an existing (“frozen”) LArSoft release
- Develop and build in LArSoft, based on the latest frozen release
- Initial setup
- When to add other repositories
- Building your code
- Setup your code to run
- Shortening the development cycle
- When a new LArSoft release is available and you want to develop against the new LArSoft release
- Useful commands
- Git protocol: Make a new branch
- hot fix
- More gitology: While you are developing…
- More gitology: Update your feature branch with updates to the repository
- More gitology: Merging your changes back into the repository
- Removing a package from your local area
- Feature branch
- More complicated development tasks
- Miscellaneous Tips, Tricks and Useful Stuff
The following is a quick guide to the things you, a member of DUNE, will need to do in order to run and develop applications in LArSoft. Please note that there are more general instructions/guidance at https://cdcvs.fnal.gov/redmine/projects/larsoft/wiki/_Quick-start_guide_to_using_and_developing_LArSoft_code_, some of which this page is ripping off! That wiki is definitely worth looking at -- perhaps even before reading this document. The following assumes
- You have all your computing privileges and accounts, including an account on the lbne gpvm nodes (e.g. lbnegpvm01.fnal.gov).
- You are working on one of the lbne gpvm nodes. Many of these should work on other machines/sites as well, if they are configured in the standard way, but there are no guarantees for that. Talk to your system/local administrator if you’re having trouble, to make sure there’s no differences in the basic setup.
- You have a working knowledge of UNIX.
- You are using the bash shell (most of the things for csh should be clear)
- You know something about the general structure of LArSoft. If you don’t know what a fcl file is, and/or don’t know how to make/modify one, then you will need a more basic tutorial than what’s here.
Run from an existing (“frozen”) LArSoft release¶
Then, to list the available versions of the LBNE software, do
ups list -aK+ lbnecode
You should see a list (possible long list…) of versions of the code to set up. Each line should look something like this:
“lbnecode” “v1_00_01” “Linux64bit+2.6-2.5” “debug:e4” “”
Pick which version you want to setup (probably the latest, and probably the profile version), and do something like this (with the version and qualifiers you want):
setup lbnecode v1_00_01 -q debug:e4
This will automatically setup larsoft as well, so don’t worry about anything else. Meaning, you don't need to also do
setup larsoft v1_00_01 -q debug:e4. lbnecode of a particular version requires larsoft of that version, and thus it sets it up too. You can think of “lbnecode” as LBNE’s own particular implementation of larsoft. You should then be able to run your usual larsoft job:
lar -c my_fcl_file.fcl ...
Develop and build in LArSoft, based on the latest frozen release¶
We advise strongly that each new directory you would like to make corresponding to a new development environment you will create should only be performed once per login. So
mkdir larsoft_mydev2 below should only happen after logging out and logging back in. The reason is that your $PRODUCTS variable tends to grow to include all working areas you visit per login. It can happen across logins too. This makes for many weird effects upon saying
mrbsetenv. Check your $PRODUCTS when confused.
For developing your code, we recommend you use the latest LArSoft frozen release. The LArSoft frozen release is tagged and built once a week. By using the latest frozen release, you get the most up-to-date features and a stable release. We have instructions below on how to update your local release when a new LArSoft release is available.
Like above, the first thing to do is
But now, things get different! First, check the larsoft releases:
ups list -aK+ larsoft
Then setup the latest version of LArSoft, e.g. (note v03_04_04 -q e6:prof is just an example, setup the latest larsoft version you find from this command. Note that the list from the ups command is NOT sorted, so the latest version is not the last one in the list)
setup larsoft v03_04_04 -q e6:prof
Another way to get the latest version of LArSoft that works with the latest lbnecode is to check this file and get the larsoft version on the line that starts with 'larsoft':
Do not use "nightly" as lbnecode nightly version is not supported at this moment.
Now, you should make a new directory where you intend to put your code (not your AFS home, as the libraries you build can get large, but somewhere beneath /lbne/app/users/USER/). Let’s say that new directory is going to be called
larsoft_mydev. Then do
mkdir larsoft_mydev cd larsoft_mydev
Then, inside the new directory, do
Do the following to use the local release:
Now, pull down
cd srcs mrb g lbnecode
When to add other repositories¶You should add other repositories if
- you need new features in that repository that were added after the last frozen release, or
- you want to develop code in that repository, or
- the current head version of lbnecode depends on the head version of another package, watch announcement.
cd srcs mrb g larreco
Building your code¶
Go to the build directory in your development area,
and then, if this is your first time building with these local packages, do
You can omit this if you have already done this, but if you at any point add a new package, you should do
mrb z; mrbsetenv. Meaning, you want to redefine the CMakeList.txt files and hence the Makefiles. Once properly set up, you can build/install:
mrb i -j4
That latter part tells it to build on 4 cores, if available. It may be easier to break that step above into two parts---
mrb build -j4; mrb install---in order to more easily check for compilation issues.
Setup your code to run¶
After everything is built and installed, do
to setup your newly built code. (‘slp’ stands for setup local products.) Now you can run your lar job:
lar -c my_fcl_file.fcl ...
Shortening the development cycle¶
Once you're stable and the code you're working on is well defined, and you're not git pull'ing new repositories, the development cycle can be shortened. Go make your edits in ../srcs/lbnecode/PackageA/AnalysisCode.cpp and jump back to ../build. You can now replace the onerous
mrbsetenv mrb i -j4 mrbslp
make install -j4
That should cut a few minute procedure to sub-minute.
The next time you login a lbnegpvm node, do the following to set things up:
source /grid/fermiapp/lbne/software/setup_lbne.sh cd larsoft_mydev source localProducts_XXXX/setup mrbslp
When a new LArSoft release is available and you want to develop against the new LArSoft release¶
- Relogin a lbnegpvm node, do the following
source /grid/fermiapp/products/dune/setup_dune.sh setup larsoft vxx_yy_zz -q e7:prof #set up the latest larsoft release cd larsoft_mydev mrb newDev -p #this creates a new localProducts_XXXX directory using the new larsoft release and the existing srcs directory source localProducts_XXXX/setup #make sure to use the new localProducts directory, it's better to delete the old directory cd srcs/lbnecode git checkout develop git pull
- If you are working on a feature branch:
cd srcs/dunetpc git checkout develop git pull git checkout feature/<my feature branch> git merge develop
- Update all other repositories (e.g. larreco)
- Make a clean build
cd $MRB_BUILDDIR mrb z mrbsetenv mrb i -j4 #rebuild
This command shows the versions of all ups packages and their locations:
Git protocol: Make a new branch¶
Now, if you want to just build, see below. However, you will also want to develop some stuff. Before touching the code, do this in the
srcs/<package> directory (e.g.
src/larreco/). If you cd into that, you will then be inside a particular repository. Git commands make sense only if you cd there.
git flow feature start $USER_testFeature
$USERis your username, and
testFeatureis up to you. You should now see that you are on this new branch (
feature/$USER_testFeature) if you issue
git branch -a. This helps you comply with the First rule: never push to master branch.. Make all of your changes, and commit as you will. You will need to create a new branch like this for every repository/package in which you are changing code! This is all done manually now, but expect nice scripts that do this for you in the future.
The reason for making your own branch and working on it is because you always need to be in a position to only commit and push a fraction of what you have. First rule: never push to master branch. You may realize some part of what you'd like to push is not ready yet. Or you may wish to suddenly fix one other totally orthogonal thing in origin/develop and push it -- a hot fix. You need to be adaptable to commit/push'ing half-measures. The easiest way to accommodate that is to be on your own new branch. Stay off your own develop branch until you know you want to push into origin develop. (See below
git fetch, rebase where that's done.) It's just a good idea.
You have now defined a local branch for you to work in. The git repository of origin as yet knows nothing about this branch. We will come back to this.
We will not pretend that there is not a short cut around all this manufacture of extra branches and git flow usage. Bare git commands can be used to make quick changes. Namely, if you know you are going to fix one piece of code in, e.g., larcore/SimpleTypesAndConstants/PhysicalConstants.h maybe there's no reason to make this extra branch. You will fix develop, and that's it. Here's how to do that. Remember the First rule: never push to master branch..cd larsoft_mydev/srcs > mrb g larcore > # make your edits, build, test the code... > cd larcore > git pull origin develop # get any git pushes that have happened since you last pulled > git branch -a > git checkout develop # if needed, to get you onto the develop branch in order to comply with the *First rule: never push to master branch.*. > git commit -a -m "Make hbar = h/2pi" # promote your changes to being ready to push > git push origin develop # push back from whence they came
More gitology: While you are developing…¶In git, you have your own “repository” in your development area. You can and should make frequent commits to it. Note that this, unlike svn, won’t change the global repository that everyone uses, for two reasons:
- because you are good at following best practices, you have made your own branch in which all of your changes are going into; and,
- you are only making commits/changes to your own local repository.
So, suppose you have changed (or created!)
my_file.cc in some
srcs/<package> area. To commit this change, do
git add my_file.cc git commit -m ‘message about the commit here; if you do not use -m, it will open a text editor and allow you to make a very long commit message’
You can add multiple files at once (
git add my_file.cc my_file.h) and use just one commit message for them. Or
git add my_dir, and it adds recursively everything in directory
my_dir. Or simply skip the
git addby doing
git commit -a -m "blah blah blah".
Also incredibly helpful is the
git status command, which will tell you all of the files that you have changed, new files that you have created, and any files you have staged---i.e., done
git add on but not yet committed.
More gitology: Update your feature branch with updates to the repository¶
It’s likely that people will make updates to the repository (
develop branch) while you are working on your feature. It’s a good idea to incorporate those regularly, and it’s a requirement that you do that before pushing in your own commits. To do that, go to the package you want to update and rebase
cd larsoft_mydev/srcs/<package> git fetch origin git rebase origin/develop
You may then need to address rebase conflicts (and once you do, commit those changes).
More gitology: Merging your changes back into the repository¶
Once you have tested your feature, and are ready for the world to use it, do the following:
cd larsoft_mydev/srcs/<package> git fetch origin git rebase origin/develop git flow feature finish
The first two commands will fetch whatever changes from origin (on all branches) that you've missed out on whilst working in your local area and shove them into your changes -- or rebase you to origin/develop. Then the
git flow feature finishwill take your feature branch and merge it back into your
developbranch (remember, you have your own repository!). Sometimes you'll see
git pullto accomplish the first two things above.
pullis roughly a combo of fetch and merge.
rebaseis a more sophisticated version of merge, which works along the time axis first, before merging -- I think. Now you need to push your changes in
developto the main (
git push origin develop
and congratulations, you've unleashed your terror on the LArSoft world!
Removing a package from your local area¶
Finally, after you’re done working on it, you’ll probably want to remove a package from your area. To do that for
cd larsoft_mydev/srcs rm -rf <package> mrb uc #Removes references to that package from the build script. rm -rf ../localProducts_XXXX/<package> #Remove product from localProducts directory.
Remember, to build again, you should do
mrb z; mrbsetenvin the
builddirectory before trying to build (just to be safe!).
Feature branch¶There are many reasons to make a feature branch:
- You are testing things that you don't want other people to use yet.
- If you are making breaking changes, e.g. you are changing code in two repositories and people need to checkout both repositories in order to compile code, it is good practice to put your changes in feature branches and ask the release manger to merge them back into develop when he/she is tagging a new release.
- You want to collaborate with other people on code development.
This page has all the useful information on git flow/feature branch:
To create a feature branch.¶
git flow feature start user_MyFeatureAddition
It is recommended to add your user name in the name of the feature branch so people know who is responsible for this feature branch.
Now if you do
you should see the new feature branch.
Now you can make changes and commit them but do not push.
To publish a feature branch for collaboration purposes¶
git flow feature publish <feature>
Now if you do
git branch -a
you should see your feature branch in the list of remote branches as remotes/origin/feature/<feature>.
To update a published feature branch after making commits:¶
Make sure you are on the correct feature branch:
If not, do
git checkout <feature>
to push your changes to the remote feature branch.
Work on an existing feature branch created by someone else¶
First make sure the feature branch is publish. Do
git pull git branch -a
to verify remotes/origin/feature/<feature> exists.
git checkout feature/<feature>
to checkout the feature branch to your local area. Do
to verify you are on that branch.
Merge develop into a feature branch¶
If you need to fetch upstream changes in develop branch:
git checkout develop # go to local develop branch git pull #update local develop branch to be in sync with remote develop branch git checkout <feature> # go back to the local feature branch git merge develop # merge your local develop branch into your local feature branch
If the last step gives errors on failed merging, open the files and resolve conflicts indicated by ">>>>>" and "<<<<<". After all conflicts are resolved, do "git commit -a"
If you want to update remote feature branch, you can do "git push" now.
Merge a feature branch into develop¶
If you want to merge a feature branch into develop and still keep the feature branch:
git checkout develop # go to your local develop branch git merge <feature>
Resolve conflicts if necessary.
If you want to push your changes to remote develop, you can do "git push" now.
Finish a feature branch¶
If you are done with a feature branch and merge it into develop
git flow feature finish
This would merge your feature branch into develop and delete the feature branch. You are on local develop branch now.
Resolve conflicts if necessary.
If you want to push your changes to remote develop, you can do "git push" now.
More complicated development tasks¶
The above should be enough to do what you are typically used to doing on a day-to-day basis, and it’s a good place to start. Once you think you have those down, there are some further things you may want to be able to do, outlined below.
Creating a new module in an existing package¶
Imagine you're working in your directory in your git clone'd, pull'd version of larsim/LArG4. LArG4 is a package -- it's a top-level directory underneath a project, which makes it a package in our parlance. Say you would like to create histograms or a TTree that characterizes some truth level information. (You're somehow unhappy with the module LArG4Ana_module.cc that already does this and want to start fresh.) You thus want an ED::analyzer module.
To accomplish this you would proceed like any right-thinking physicist and cp another *Ana_module.cc file from somewhere else and pull out all the un-needed bits, leaving the key structure of the module -- the methods you must over-write to make this thing work. You would then declare, initialize, and Fill your histograms/TTrees in the right places. (You don't need to Write() them because the TFileService in ART takes care of that, as usual.)
Now just go back to your build directory and
mrb z; mrbsetenv; mrb i -j4. You hopefully don't need to touch any of the files that do the building, because the drilling down to find your new .cc just does this as a feature in the CMakeLists.txt. Now, if you had added stuff that needs access to header files that were not already included in larsim/LArG4/CMakeList.txt you would need to edit that file to add them. Same if you added new functions/methods not already in Libraries in that CMakeList.txt.
Creating a new package in an existing repository¶
Okay, instead, let's add a new directory under the existing repository, itself containing one or more modules and perhaps other .cpp and .h files. First, to get it to compile, your Makefile needs to know about it. As yet, it does not know about it. Remember, the cmake that happens in the first layer of doing an
mrb b or
mrb i takes your CMakeLists.txt files and creates the Makefiles. So that's what happened above, and merely adding a file in an existing directory, as we did there, did not complicate matters too much. Here, we have at least one other step. We'll need to inform the top level CMakeLists.txt that you have a new directory into which to drill down. You'll then need a CMakeLists.txt in that directory too.
Let's do this in lbnecode. Add a new add_subdirectory(MyNewDirectory) line in lbnecode/lbne/CMakeLists.txt. That is, you are directing it to drill down into lbnecode/lbne/MyNewDirectory to look for the next CMakeLists.txt file to generate the Makefile to build all the fabulous code you've put there. cp a CMakeLists.txt file that is nearby. Use the cetbuildtools macros already in there and from other CMakeLists.txt files to specify any new include directories to compile against and libraries to link against.
Creating a new repository/product¶
In fact, we don't see too common a need for this from the lay-collaborator. Imagine coming along with a whole new package of code (which itself might live in nusoft, say) for which you've written specific algorithms, e.g. Pandora. That use case is one for code that might merit a whole new repository. But, any new wire simulation or optical scintillation package or calorimetry reco module can go into an existing repository: lbnecode or larsim or larreco, correspondingly.
If you do need to do this you also need all the "hair" that comes with the package to make it build-able and ups-able by the cetbuildtools system. We leave details to https://cdcvs.fnal.gov/redmine/projects/larsoft/wiki/_Quick-start_guide_to_using_and_developing_LArSoft_code_#Creating-an-entirely-new-product-within-an-existing-work-area.
Multiple build and install areas.¶Use multiple install areas when you want to build flavors using different ups qualifiers (-q setup option). This method would typically be used when compiling your code using debug compiler options (-q debug:e4) vs. profiled/optimized compiler options (-q e4:prof). Each install area should be initialized using the command
mrb newDev -v v1_00_04 -q debug:e4 -T debug.v1_00_04 -f mrb newDev -v v1_00_04 -q e4:prof -T prof.v1_00_04 -f
mrbto put the build and install areas into the specified subdirectory. The
mrbthat it is OK to use an existing
sourcearas. Here are a couple of caveats about this method.
- You can omit the -v and -q options if you have
mrbwill inherit the version and qualifiers from your setup version of
- Your specified qualifiers are hardwired into the generated
localProductsXXXX/setupinitialization script. This initialization script is not reentrant. That is, you can not switch flavors simply by sourcing a different
localProductsXXXX/setupscript. You should always source
localProductsXXXX/setupin a fresh shell.