Project

General

Profile

Qualifiers in cetbuildtools

See also About qualifiers, which contains the usage of qualifiers as of this writing.

See also the Day 2 qualifier discussion notes.

This has been implemented. As we discovered the various things which needed to be specified, this evolved into a single product_deps file, as explained in Defining products in the CET build environment

Purpose

This is a record of decisions (and proposals) regarding how cetbuildtools will make use of UPS qualifiers and relationships between products to help ensure consistent builds of products we create.

Product versions

The version of a product indicates the source code used to create the product. cetbuildtools has the ability to make several different installation tarballs from the same source text (e.g. by using different compiler versions); two installation tarballs of art which claim to be version 1.00.10 shall always have been built from the same source code. In products for which we control the source code, the version number shall correspond to a tag in the code repository.

It is allowable (though it should be rare) for two different repository tags to be applied to the same source code, and thus for two different product versions to actually have been built from the same source text.

Each binary distribution tarball for a given product shall be created from a single version of that product (and thus a single tag of the repository, if we control it).

Binary distribution tarball qualifiers

The value of the qualifiers particular to a specific binary distribution tarball imply the values of the qualifiers of all of all the lower-level products' distribution tarballs upon which the top-level product depends. Note that we say here that the binary distribution tarball has these qualifiers, rather than the product having these qualifiers, or the product version having these qualifiers. That is because the same product may have many different versions, and the same product version can be made with many different qualifiers. It is actually the result of the build, which we deliver by a binary tarball, and which one can obtain access to through use of the UPS setup command, that has these qualifiers.

The qualifiers, as understood by cetbuildtools, are merely strings; cetbuildtools understands no semantics.

However, we propose that our best practice recommendation to others, and the rule we will follow ourselves, is that qualifier names carry up to three colon-delimited parts. Two of these standard parts reflect our understanding of what is needed for consistent builds of libraries written in C++ for use of other projects written in C++. They are likely not to be needed for C code. We make no pretense of having analyzed the problems of building code in other languages. These two parts are:

  1. The "optimization level", which is one of debug or prof or opt
  2. The "almost-ABI specification", which denotes the compiler used and enough of its version information to specify that which the compiler vendor claims to be ABI compatible. For GCC, this would be something like gcc46 or gcc46e (where the e indicates C++0x mode; we treat this as a different ABI).
The third optional part of the qualifier is an arbitrary string whose purpose it is to convey all other relevant information about the built product. This information includes:
  1. Any optional products included in the build of the product being described (e.g. Geant4 can be built using an internal CLHEP or an external CLHEP; the qualifier of a Geant4 build would indicate which is the case).
  2. The qualifiers of each product upon which the product being described depends (e.g. art can be built against a Root build that includes support for Geant4 or against a Root build that does not include support for Geant4; the qualifier of the art build which specify which qualified Root build it depends upon, which in turn defines whether that build of Root does or does not include support for Geant4).
  3. Whether or not the build includes other extra features which do not affect the ABI, such as whether the product was compiled with -fopenmp (to support use of OpenMP) or not.

The reason for separating the first two items from the general case supported by the third is the convenience of the non-expert user. For many of our lower-level products, it turns out that no third qualifier is needed. For such products, a simple ups list of the product database will tell the user all the "version information" of interest.

For the product that are more complex, and which require the third qualifier component, the interpretation of that qualifier component would require looking at the file in the source distribution that contains the qualifier-related information.

Because no semantics of the first two qualifier components are known to the build system, we have the flexibility, if needed, to "fake out" the system in cases where we determine it to be to our advantage to do so. For example, it seems to be the case that a gcc46e version of art can be built upon a gcc46 version of Root; we are currently doing this, and would like to retain the ability.

Where the information resides

A single installed version of a product can contain multiple "instances", which differ by platform and by qualifiers. Each installed version of a product contains a single UPS table file that must contain information about all the different instances of that version of the product. We propose that these table files be automatically generated to contain all interesting combinations (as defined below), even when we do not actually build an installation tarball for a particular qualified instance. We propose that these table files be automatically generated by cetbuildtools, and thus that they are not the original source of dependency information, directly maintained by the author of a product.

We propose the author of a product be responsible for maintaining, in the source code repository for the product, two files which combined contain all the relevant dependency information.

In the first file, called product_deps, goes a table of the product names and versions (not the qualifiers) upon which this version of the product in question depends. Thus the source text, in the code repository, contains the information about what product APIs the given version of the product in question depends upon. The file format is a simple whitespace-delimited text file; a fictitious example for a simplified art product is given below:

product version
root 5.30.02
clhep 2.1.2.1
boost 1.47
In second file, called qualifier_deps, goes a table that tells what the qualifiers of the product in question mean, in a whitespace-delimited text format. This table contains one row per valid qualifier for the product in question, and multiple columns:
  1. one column ("qualifier") specifying the value of the qualifier being described by this line of the table
  2. one column ("notes") that describes what set of features are particular to the product in question, e.g. it was build in the "external CLHEP" mode or in the "internal CLHEP mode" (Geant4 does this) or is built with -fopenmp (where lack of a note to this effect means that the product was build without this flag, and thus without support of OpenMP). It is possible that this column is empty.
  3. one column for each dependent product listed in the product_deps file, specifying which qualifier for the dependent product is connected to the qualifier specified by that line of the table for the product in question.
  4. No column except notes may be empty.
  5. If the product has no qualifier, the column entry is -nq-
  6. If the product is not part of the dependency tree for this qualfier, a single dash is used.
  7. The notes column must be at the end. Then, it is just "from here to the end of the line" and may contain whitespace.

A fictitious example of a qualifier_deps file extending the example above is given below:

qualifier root clhep boost sqlite3 notes
F15:gcc46:debug P3:gcc46:debug P3:gcc46:debug gcc46:debug debug
Tornado:gcc46e:prof MR1:gcc46e:prof MR1:gcc46e:prof gcc46:prof - -fopenmp
Note that in this example:
  1. We have described a build of art with a C++11 compiler that uses a Root compiled with a C++03 compiler, but CLHEP and Boost built with the C++11 compiler. Presumably, we have checked that this Frankenstein assembly works.
  2. We have described a build of art that uses OpenMP that uses lower-level products that know nothing of OpenMP
  3. We have been careful to assure that each line of the file puts together things built with the same ABI and debug level
  4. We have taken advantage of the fact that sqlite is a product written in C, and so the platform-dependent C libraries are the only ones we have to worry about, and thus may need no qualifiers.

The cetbuildtools system is responsible for creating the UPS table file, and the setup_for_development script, from this information.