CET build and packaging system working session

We are currently planning for a mini-retreat, of 2 days duration, to make some important steps forward in the status of cetbuildtools, in the way cetbuildtools is used in the development of the "art suite" of products, and to make it easier to use cetbuildtools for other projects (for new experiments, and for other projects in which our group is involved) . We currently have a system that is pretty robust for expert use. We want to move in the direction of even more robustness, and especially in the direction of greater ease-of-use for non-experts.

Summary of retreat goals

We want to get our build system and packing tools into a condition that makes them easy enough to use that new experiments will prefer to use our tools rather than to build their own. Furthermore, existing experiments may be willing to move from their existing build systems to cetbuildtools. We also want it to be easier to produce a new release of art, so that all the burden of building new release no longer falls on a few experts.

We have several analysis tasks to perform. Based on the result of these analyses, we will then have some development work to be done on the various parts of our system.
The analysis tasks are listed in the subsections below.

Clarify our use of qualifiers

  1. exactly what should be meant by e1?
  2. what is the relationship between qualifiers for packages in a dependency hierarchy, i.e. how is e1 for art related to e1 for messagefacility?
  3. how will we maintain the data regarding the meanings of and dependencies between the qualifiers? Is there a need to do this? If there is a need, what is the simplest adequate solution?

"metadata" that describes the build

  1. Make sure we have all the relevant information for describing a build encoded in the system? (For example, where is the "-fopenmp" nature of a build encoded?)
  2. Is each piece of information encoded in exactly one place?

Version identifiers for dynamic libraries

  1. What version information will art make use of and track for the user?
  2. What will art make use of and track for the user? It must be possible for build systems other than cetbuildtools to insert the appropriate information.
  3. How will cetbuildtools help make sure this information is present in libraries it builds?

The build system, as viewed by a developer

We want experiments to be able to use cetbuildtools as a tool for developers. cetbuildtools already contains the buildtool script to help overcome some perceived insufficiencies in CMake (specifically the wish from mu2e to be able to use glob patterns for specifying what compilation units to process; and safety issues associated with the need to occasionally clear a build area completely). We need to make sure that "development builds" (that is, not "official releases") created by collaborators have libraries that are marked in some way as "not official". It is not necessary for cetbuildtools to keep track of versions for these "not official" libraries.

The procedure by which a user causes his build of product C to depend on his private build and installation of lower-level product B, while both depend on a official release of a still lower-level product A must be easy to follow. The system we have now seems adequate for experts, but we need something that requires less expertise.

It must be easy to generate / customize the CMake fragment that will be invoked by find_package or find_ups_package(...) -- it must be easy to add the correct include and link directives into a build to enable it to use a particular lower level product (and in turn, its lower level dependents).

The manipulation of UPS table files requires significant expertise. We should investigate what tools we can produce to make it easier for non-experts.

Given that my current working directory is a cetbuildtools working directory, how do I determine the configuration of this directory? Some things of interest (maybe there are more):
  • how do I determine what packages this build depends upon, and what their versions are?
  • how do I determine whether I have a debug or profile or optimized build?
  • how do I determine what qualifiers my build will have?

The system will also need clear documentation, including how to use it in various scenarios.

In addition:
  • Do I (as a developer) want more macros to do what I want in a CMakeLists.txt file (eg build library / exec, install library / exec, etc, etc)?
  • Are the existing macro facilities usable and sufficient for what they provide (install_headers, art_dictionary, build_plugin, cet_test)?
  • What do I need from a setup script / command to use my software (eg run an installed ART or other exec with modules installed either centrally and/or privately) vs one to develop a particular package (and easily develop a new package in the same shell without having to jump through hoops)?

The build system, as viewed by a release manager

A release manager's primary goal is taking specified versions of the source code, and building the distribution tarballs (or directly installed products?) for a specific platform or set of platforms.
A potentially useful option is building directly installed products. If the we recommends against this, then we should document that recommendation and the reasons for it. In either case, it would be nice to have some small amount of documentation listing the pros and cons of building tarballs vs. directly installed products. (The goal would be to provide guidance for someone from a new experiment in deciding how they should handle deployments.)

Release managers may be responsible for running the test suites for the installed products; if so, this must not require detailed expertise with the tests.

An important difference between the libraries built by release managers and those built by "users" is that the libraries built by release managers must have version information.


This effort has a few different phases. The requirements work needs to be done before the documentation and implementation, to avoid wasted effort. The documentation and development may be done in parallel.

  1. Completion of requirements. Most of the notes above address this effort. Estimated time: 2 days, 4-5 people (Kurt, Ron, Marc, Jim, Chris, ?)
  2. Documentation of the system. This includes both task-oriented documentation (describing how to use the various pieces to do the important 'use cases') and man-page type documentation for individual tools. Some (maybe much) of the man-page type documentation already exists. 2 days, 1-2 people (Gennadiy, Walter)
  3. Implementation of remaining pieces of the system, to meet the requirements from (1) above: 2-3 days, 3 people. (Chris, Lynn, Marc)

It is most important for the first item to be completed as a unit by a group of people. The other can happen afterwards, and not necessarily as a "sprint".

Environment for the retreat

We'll be having the requirements session (the retreat) on Tuesday and Wednesday, Feb 21 and 22.

  • Tuesday: WH 1 North
  • Wednesday: WH 8 XO (Hornet's Nest)

Agenda for the requirements session.

A great deal of requirements analysis has already been done, leading to the design and development of the current version of cetbuildtools. The requirements analysis remaining involves refinement of the system; we're looking for what needs to be easier to do, or where we've missed something.

Day one

We'll begin with a brief introduction to the reason why we did not adopt a previously-used build system, and have created our own. It is important that we understand how the build and delivery systems we have used in previous experiments and projects have not met our needs.

To help identify such things, we propose to take the first day to go through a series of "use case" exercises. We should do this in pairs, with one member of each pair being someone relatively new to (but not completely unfamiliar with) the use of cetbuildtools. This person should be the "hands-on" driver of the exercise. The other member should help answer questions, and take notes about what its are confusing, hard, or inconvenient. We may need to split up the list of "use cases", having a different pair take on different tasks, so that we can complete this part of the exercise within one day. We'll want to have Chris around to answer questions, deal with emergency fixes if needed, and to observe both teams and record more observations about what seems to be hard (where the teams are not using the tools as he would have expected, for example). If this can be completed in one morning, then we can start on the "day two morning" tasks the afternoon of the first day.

Day two

The second morning, we'll collect all the notes, and jointly compose a set of changes with think are needed. The afternoon of that day we should spend discussing the other items in the list of issues above (the items under #"Summary of retreat goals").

List of "use cases"

  1. Starting from a fresh git clone of an existing product (e.g. fhicl-cpp)
    • build fhicl-cpp and run its tests; add a new accessor to ParameterSet and write a test for the new addition
    • make private installation of this enhanced fhicl-cpp
    • make a public distribution tarball of this modified fhicl-cpp, with a new version number, etc.
    • add a dependency to another existing product (e.g. clhep; give ParameterSet the ability to return a random number or something like that...)
    • make a new private installation of fhicl-cpp
    • make a new public distribution tarball of fhicl-cpp
  2. Starting from scratch, create a new product with no initial external dependencies (if a repository is needed, we can create one on oink and eliminate it afterward)
    • Create a dynamic library with at least one function or class, with no external dependencies
    • Create and run a test executable that exercises that function or class
    • Create a non-test (installable) executable that uses the function or class from the library
    • Add a function to the library that has an external dependency on a product we build (e.g. Boost regex from Boost 1.47)
    • Create and run a test executable for the new function
    • Add use of this function to the installable executable
    • Create a personal installation of the new product
    • Change the dependence on Boost regex from Boost 1.47 to Boost 1.48
    • Create another personal installation of the product
  3. Starting from a fresh checkout of art
    • Modify some code that uses a ParameterSet to require a new feature: call std::string ParameterSet::name() const (which does not exist)
    • Check out fhcl-cpp, and add this function to ParameterSet
    • Make a local installation of the new fhicl-cpp, with a new version identifier
    • Make your art development build use your new fhicl-cpp, and verify it now can use the new function ParameterSet::name().
    • Make a local installation of your new art.
  4. Starting from scratch, create a new product that depends on art (if a repository is needed, we can create one on oink and eliminate it afterward)
    • Make a class that is initialized from a ParameterSet
    • Write an execute a test that verifies the initialization
    • Write an installable executable that runs the function artapp.
    • Write a test that exercises this installable executable
    • Create a local installation of your new product
    • Create a public distribution tarball of your new product
  5. Migrate an existing project from using something else to using cetbuildtools. (Details to be clarified).

(to be filled out)

  1. have everything carry out a few scenarios of system use - record defects and difficulties (maybe assign use cases to people)
  2. analyze the options and methods for using the system and do something with them.
  3. exam other thing ...

Last minute work entries

It would be nice to have a testing tool to be used by package developers, to make sure that the unsetup command really cleans everything up that should be cleaned up. For example, a setup x followed by unsetup x should result in no change in the user's environment. Is this achievable? Is it sufficient, or do some products do more than modify the environment?

Notes From the Working Sessions

On starting work.

  • For the purposes of the fest, we are working on oink.
  • Source /products/setup: this is where products are installed centrally on oink.
  • Prepare a local product install area:
    mkdir <path>/products
    tar xvf ~distro/served_files/private-products-setup/upsfiles-dir.tar.gz -C <path>/products
  • Set some useful environment variables so that the private install area will be used:
    export CETPKG_INSTALL=<path>/products
    export CETPKG_J=<some-small-number> # For parallel builds.

On checking out and building an existing product.

  • Check out the package from the repository, eg:
    git clone ssh://
  • In another directory (your build area), source that package's setup script:
    source <source-area>/ups/setup_for_development -d
  • Build that package:
    buildtool -tl
    The -t option specifies to build and run tests. The -l option specifies to put the verbose build output to a log file and only put "stage" information to the screen. See buildtool --help for details.

On making a new or existing non-cetbuildtools project use cetbuildtools.

  • Look at $CETBUILDTOOLS_DIR/example/ToyCmake and extract / rename / localize as necessary.

Useful links.

cet build-packaging work - day one
cet build-packaging work - day two
Day 2 qualifier discussion
Kurt and Marc notes
use case 3 notes
use case 1 notes
Gennadiy and Adam notes
Lynn's cetbuildtools WG notes
cross-compilation notes
qualifiers in cetbuildtools