Old LArSoftWiki » History » Version 43

Erica Snider, 12/01/2013 09:11 AM

1 41 Erica Snider
If you are looking for the legacy svn-based LArSoft site, see If you are looking for the legacy cvs-based LArSoft site, *all content has been moved to "LArSoft cvs (legacy site) ":*
2 25 Erica Snider
3 1 Brian Rebel
4 32 Erica Snider
5 1 Brian Rebel
6 41 Erica Snider
This is the beta LArSoft redmine project and the future home of the LArSoft redmine project.
7 29 Erica Snider
8 1 Brian Rebel
h1. LArSoftWiki
9 1 Brian Rebel
10 31 Erica Snider
*Under construction...*  Will go live when the migration to git/cmake is completed.
11 31 Erica Snider
12 1 Brian Rebel
13 19 Brian Rebel
The LArSoft software is designed to work for all planned and running liquid argon experiments at Fermilab. It is written in C++ and built on the "ROOT": data analysis software and the "FMWK": framework for HEP experiments. The releases of the software are managed using an "SRT": distribution.
14 1 Brian Rebel
15 17 Brian Rebel
To join the LArSoft mailing list, please follow these "instructions": using the list name LARSOFT.
16 32 Erica Snider
17 32 Erica Snider
18 35 Erica Snider
h1. Getting started
19 1 Brian Rebel
20 35 Erica Snider
h2. Access to Fermilab computing
21 1 Brian Rebel
22 35 Erica Snider
h3. Load balanced access to GPCF VMs
23 1 Brian Rebel
24 35 Erica Snider
h2. Where to find the software
25 1 Brian Rebel
26 35 Erica Snider
At Fermilab, the software lives in a set of re-locatable ups products, each of which corresponds to the code within a git repository. Each product and associated repository contain LArSoft software components (i.e., SoftRelTool "packages") that are at a similar layer of functionality. The reference products and repository urls are the following:
27 35 Erica Snider
28 35 Erica Snider
29 35 Erica Snider
30 35 Erica Snider
name* | *repository url (all in Redmine)* | lxr link 
31 35 Erica Snider
                                            (not yet avail) | Redmine browser |
32 35 Erica Snider
|larcore| ssh:// | -- | 
33 35 Erica Snider
|lardata| ssh:// | -- |
34 35 Erica Snider
|larevt | ssh://  | -- |
35 35 Erica Snider
|larsim | ssh://  | -- |
36 35 Erica Snider
|larreco| ssh:// | -- |
37 35 Erica Snider
|larana | ssh://  | -- |
38 35 Erica Snider
|lareventdisplay| ssh:// | -- |
39 35 Erica Snider
|larexamples|     ssh://     | -- |
40 36 Erica Snider
|larsoft | (A product that exists only to set up the others with a single 
41 36 Erica Snider
           command. The larsoft product otherwise has no content.) | -- |
42 1 Brian Rebel
43 36 Erica Snider
(The SoftRelTools-based packages in each product/repository can be found "here": along with the approximate order of dependency.)
44 1 Brian Rebel
45 36 Erica Snider
You can use the @git clone <repository url>@ command to download a copy of each repository to your local area. Additional steps are needed to use, build or develop the software. These steps are described in the "How to use and develop LArSoft code" section below.
46 35 Erica Snider
47 35 Erica Snider
48 33 Erica Snider
h2. Accessing LArSoft
49 32 Erica Snider
50 33 Erica Snider
* Repositories
51 1 Brian Rebel
* Browsing the software
52 1 Brian Rebel
53 1 Brian Rebel
h2. Release notes
54 1 Brian Rebel
55 1 Brian Rebel
| *Release* | *Date* | *Purpose* | *Changes / notes* | *Full release notes* |
56 42 Erica Snider
| v1.00.00| Jan 2014 | First production 
57 42 Erica Snider
                       release | Replica of final svn-based release. Future LArSoft development 
58 42 Erica Snider
                                 proceeds from this release. | xxx |
59 42 Erica Snider
| v0.0x.0y| 12/02/2013| "beta" limited release | Beta suitable for expert testing | N/A |
60 35 Erica Snider
| v0.00.09| 11/25/2013| "beta" pre-release | Second full release under new system. First full re-factoring
61 35 Erica Snider
                                       of experiment-specific and core LArSoft code in the larcore,
62 35 Erica Snider
                                       lardata, larevt, and larsim products. Preparation for expert
63 35 Erica Snider
                                       user testing of beta release.| N/A|
64 42 Erica Snider
| v0.00.04| 9/15/2013| "alpha" release | First release of git/cmake/ups-based LArSoft products
65 42 Erica Snider
                                       Used for mrb, configuration and re-factoring
66 42 Erica Snider
                                       development and testing | N/A |
67 33 Erica Snider
68 1 Brian Rebel
h1. Documentation
69 32 Erica Snider
70 43 Erica Snider
* LArSoft
71 1 Brian Rebel
72 43 Erica Snider
* Quick-start guide to using and develop LArSoft code
73 1 Brian Rebel
74 43 Erica Snider
* Detailed documentation on using and developing LArSoft code
75 1 Brian Rebel
76 43 Erica Snider
77 43 Erica Snider
h3. Overview of the user and developer environment
78 43 Erica Snider
79 40 Erica Snider
LArSoft releases are distributed via "re-locatable" ups products. (A "re-locatable" ups product has a simplified structure compared to that of legacy ups products, and does not require that it be "declared" to a ups database. These features simplify distribution, installation and maintenance of re-locatable products.) People interested in using the core LArSoft software, but who have no need to modify or develop that software can in principle simply perform the appropriate ups @setup <product> <version> -q <qualifier>@ commands, then build their code against those products by reference to the corresponding $<PRODUCT_NAME>_INC and $<PRODUCT_NAME>_LIB environment variables. In addition to the individual products, there is a "@larsoft@" product that can be used to set up all other products with a single command:  
80 36 Erica Snider
81 40 Erica Snider
setup larsoft <version> -q <qualifiers>
82 36 Erica Snider
83 36 Erica Snider
The versions and qualifiers available can be obtained by using the following command:
84 36 Erica Snider
85 36 Erica Snider
ups list -aK+ larsoft
86 1 Brian Rebel
87 43 Erica Snider
The qualifiers will be one of the following: "debug:e2", "prof:e2", "opt:e2", where:
88 1 Brian Rebel
89 36 Erica Snider
* debug = debugging symbols available
90 43 Erica Snider
* opt = compiler optimizations enabled, no debug symbols
91 36 Erica Snider
* prof = compiler optimizations enabled, profiling code generated
92 43 Erica Snider
* e2 = built with gcc 4.7.1 and -std=c++0x
93 36 Erica Snider
94 43 Erica Snider
In general, only @debug@ and @prof@ versions will be provided, since there is no run-time performance penalty for code that has the profiling code present.
95 41 Erica Snider
96 43 Erica Snider
The recommended way to work with LArSoft is to use *@mrb@*, the multi-repository build tool, to check out and build your own code. This build system is based upon *@cmake@* and a toolkit of @cmake@ macros in the *@cetbuildtools@* product -- the same set of tools that are used to build the @art@ framework that underpins LArSoft. @mrb@ operates above @cmake@ and drives the build procedure across (possibly) multiple repositories within one's working area. Using @mrb@ ensures the integrity and structure of the working. 
97 38 Erica Snider
98 43 Erica Snider
One's working directory within this system has the following structure:  a source code sub-directory where all development takes place; a build sub-directory where all build activities take place; and a local products area, where all successfully built software is installed, and from which it can be run.
99 43 Erica Snider
100 43 Erica Snider
All software packages built and installed by @mrb@ / @cmake@ / @cetbuildtools@ are in the form of a re-locatable ups products. @mrb@ provides a simple product template that includes two files that must be modified by the user:  a top-level @CMakeLists.txt@ file, that specifies which sub-directories will be included in the build along with any global configuration needed to build the product; and a @product_deps@ file that specifies the dependencies for the package. The product template can otherwise be customized for an experiment. Although this scheme may sound cumbersome and arcane, it is essentially no different than following the package structure conventions imposed by SRT, using a GNUmakefile to configure the build, and a global release setup script to configure the global environment. With @cmake@, the GNUmakefiles are replaced by @CMakeLists.txt@ files, while the "global environment" for the product is managed locally with the @product_deps@ and top-level CMakeLists.txt files for the product. @mrb@ then manages all the details of utilizing ups to configure the build environment, driving the build, and packaging and installing the ups product in the local products area.
101 43 Erica Snider
102 43 Erica Snider
103 1 Brian Rebel
104 41 Erica Snider
There are several common scenarios under which most people using LArSoft will be working. Details of how to operate within several of these will be provided on a separate page. The intent here is only to give a summary of some of the basic steps and philosophies that come into play in one of the most common.
105 1 Brian Rebel
106 41 Erica Snider
Let's first assume that a user has no need to modify or develop LArSoft code, but wants to work on existing experiment or personal software that depends upon LArSoft. The basic workflow goes something like the following:
107 1 Brian Rebel
108 41 Erica Snider
# Create a new working directory, use @mrb newDev@ to create an empty skeleton of an @mrb@-based working area inside. This working area will have three basic areas:
109 41 Erica Snider
** A @srcs@ sub-directory that will contain all the source code. Repositories are cloned or packages are checked out into this area.
110 41 Erica Snider
** A @build@ sub-directory, where all build activities take place.
111 41 Erica Snider
** A @localProducts@ sub-directory, where all built products are installed. Users run against locally built code that is installed in the @localProducts@ area. A setup script provided by @mrb@ can be used to place this area at the front of the ups PRODUCTS path. 
112 41 Erica Snider
# Check out the desired software and create a local copy in the @srcs@ directory using the @mrb gitCheckout@ or @mrb svnCheckout@ commands.
113 41 Erica Snider
# Modify the code as desired, checking in as appropriate
114 41 Erica Snider
# Move to the @build@ area, configure the build environment using @source mrb setEnv@, then build the code using @mrb build@
115 41 Erica Snider
# Install the built code in the @localProducts@ area using @mrb install@. Note that further development and builds have no effect on the code installed in the @localProducts@ area. A user can continue to use the installed code while simultaneously fixing an error that breaks the build in the same working area.
116 41 Erica Snider
# Set up the newly installed products using the appropriate @ups@ @setup@ commands (e.g., @setup <product_name> <version> -q <qualifiers>. For user code, the product name, version and qualifiers can be set to (almost) arbitrary values.) At this step, any global product instances will be overriden by the local versions.
117 41 Erica Snider
# Run the code from a convenient location. The top-level directory in the working directory is an appropriate choice.
118 41 Erica Snider
# When ready to commit, use @git add@, @git commit@. Bring in any developments on the head using @git pull@ or @git fetch@ and @git merge@, as appropriate
119 41 Erica Snider
# Perform final tests
120 41 Erica Snider
# Perform final commits and merges into develop, then @git push@ the results back onto the develop branch of the reference repository.
121 41 Erica Snider
122 41 Erica Snider
123 41 Erica Snider
Another common scenario will involve the type of user (not modifying or developing LArSoft), but who now needs to start from scratch to create a new product. (This could by any new code, even code that is intended for later integration into an existing product or repository.) In this case, a couple of new steps need to be added near the beginning in order to create an empty product skeleton within the @srcs@ area. The first part of the workflow then looks like this:
124 41 Erica Snider
125 41 Erica Snider
# Create a working area and working area skeleton as before using @mrb newDev@.
126 41 Erica Snider
# Create an empty product skeleton in the @srcs@ area using @mrb newProduct@
127 41 Erica Snider
# Attached the product area to an existing git (or svn) repository using @git remote add@ and @git pull@. Note that the repository could be empty at this point.
128 41 Erica Snider
# Develop code on the "develop" branch, or use @git flow" to create a new feature branch on which write the new code. Use @git add@ as needed.
129 41 Erica Snider
# Modify the top-level CMakeLists.txt file created for the product by @mrb newProduct@
130 41 Erica Snider
# Modify the @ups/product_deps@ file to include all dependencies.
131 41 Erica Snider
# Build the code in the @build@ area using @mrb build@
132 41 Erica Snider
# Install the code using @mrb install@
133 41 Erica Snider
# Run from the top-level work area directory.
134 41 Erica Snider
# Test, commit, merge and push as described above.
135 41 Erica Snider
136 41 Erica Snider
137 41 Erica Snider
h3. Overview of the LArSoft developer environment
138 41 Erica Snider
139 41 Erica Snider
The situation for the LArSoft developer is essentially the same as that of general users, except that the code that is checked out and modified will be that belonging to one or more of the LArSoft git repositories, possibly (perhaps usually) in addition to experiment-specific code. One consequence of this is that developers are expected to follow a specific development workflow in order to maintain the integrity of the main "develop" branch. Git offers many features and capabilities that make it extremely easy to isolate one's work from the rest of the world, to preserve it in the reference repository and to share it with others without affecting the reference develop branch, while at the same time following recent developments in the reference develop branch or in other branches. Consequently, there is little need to push changes to the reference develop branch before they are known to integrate into the build properly.
140 41 Erica Snider
141 41 Erica Snider
The LArSoft project has adopted @git flow@ to assist with managing the development workflow. Within this framework, the git repositories have the following branch structure:
142 41 Erica Snider
* A "main" branch that will have only tagged releases. Used only by the software manager.
143 41 Erica Snider
* A "develop" branch that will have the working head of the repository. Used by all developers.
144 41 Erica Snider
* A "release-*" branch for the integration of specific tagged releases. Used or authorized only by the software manager.
145 41 Erica Snider
* An arbitrary set of "feature" branches on which all on-going work takes place. In most cases, these branches will be in local repositories, although publishing them to the reference repository is allowed. Developers can create as many feature branches as needed. 
146 41 Erica Snider
* A "hotfix" branch that is used to develop patches to tagged releases. Used or authorized only by the software manager. 
147 41 Erica Snider
148 41 Erica Snider
149 41 Erica Snider
150 41 Erica Snider
151 41 Erica Snider
152 41 Erica Snider
153 41 Erica Snider
h2. Detailed workflow notes
154 32 Erica Snider
155 32 Erica Snider
h2. Tools for working with the software
156 32 Erica Snider
157 32 Erica Snider
*  git
158 32 Erica Snider
*  git flow
159 1 Brian Rebel
*  mrb : the multi-repository build tool
160 39 Erica Snider
*  Re-locatable ups
161 39 Erica Snider
*  cmake
162 32 Erica Snider
163 32 Erica Snider
164 32 Erica Snider
165 32 Erica Snider
* Basic workflow
166 32 Erica Snider
* Using git
167 32 Erica Snider
* Multi-repository build
168 33 Erica Snider
169 33 Erica Snider
170 33 Erica Snider
h1. How to
171 33 Erica Snider
172 33 Erica Snider
173 33 Erica Snider
h2. Advanced technical how-to's
174 33 Erica Snider
175 33 Erica Snider
h2. Release procedures
176 33 Erica Snider
177 33 Erica Snider
h1. Walk-through exercises
178 34 Erica Snider
179 34 Erica Snider
180 34 Erica Snider
h1. Working areas
181 34 Erica Snider
182 34 Erica Snider
[[Beta re-factoring]]