Project

General

Profile

[ Please note: this page remains for historical purposes only: it has been entirely superseded by the document Software product building and management requirements ]

Requirements for an externals build system

The purpose of the externals build system is the building and packaging of UPS products from source code we do not control, and which use build systems not under our control.

Note: if one were to imagine a system that does not use UPS, then the requirements on that system would omit UPS, but incorporate in lieu the requirements that are satisfied by UPS, including (but not limited to):
  • Side-by-side installation of multiple versions and variants of the same package.
  • Ability to relocate installed packages and package sets.
  • Management of package dependencies.
  • Ability to use packages installed in multiple locations, with duplicate resolution.
  • Ability to install packages without system-level privileges.

The requirements for a full-featured externals build system:

  1. The build system must be able to operate on all the platforms and architectures we support. This does not include running on platforms for which we will cross-compile code, but does include the ability to run on the host system from which we will do the cross-compilation.
  2. The system must be capable of producing relocatable UPS packages as an end product.
  3. The user must be able to describe, as a self-contained entity, the instructions for building a particular package; the system must be responsible for locating and executing the build instructions for necessary dependencies.
    The configuration for a particular package should be sufficient to specify its dependencies; the user should not have to further specify the full list of interdependent configurations, otherwise there is scope for inconsistency and error.
  4. The user must be able to specify a group of not-necessarily-related packages to build as a set.
  5. A user must be able to describe a package that consists only of dependencies (i.e. an "umbrella package").
  6. The system must be capable of building a series of multiple packages in the correct order as specified by their dependencies.
  7. The system must be capable of skipping already-built packages, and of forcing a rebuild starting at a particular point in the sequence.
  8. The system must be capable of rebuilding a particular package, checking for the presence of dependencies and (at the user's option) either recursively building those dependencies or failing due to missing dependencies.
  9. The user must be able to easily update version numbers of dependencies of an updated package, within a defined, coherent set, when required. For example, when the version of SQLite is updated, it must be easy for the user to update the versions of all packages that depend upon SQLite (Python, and everything that depends upon Python).
    The more manual steps involved in this process, the more there is scope for inconsistency and error.
  10. The system must be capable of building multiple distinguished configurations ("qualified builds", in UPS parlance) of the same package and version (compiler, feature set, etc.), with dependencies specified including version and qualifiers. It should not be assumed that consistent products will have identical qualifiers (an "e4" build of a C++-based product may rely upon an unqualified build of a C-based product). Note that this includes the ability to control what compiler and what version of that compiler is used, including the use of cross-compilers.
    Packages, depending on their source language, may have different dependencies on a particular compiler or set of options (C++2011, for example). A given "set" of packages (e.g. Mu2e) may have different variants (e.g. debug vs prof, GCC 4.8.1. vs GCC 4.8.2 vs ICC) that should be propagated down through dependencies.
  11. The user must be able to describe dependencies between packages that differ for different versions, qualifier-sets or platform.
  12. The system must be able to generate the necessary UPS table files. The specification of dependencies must be done in only one place. The specification of what is built and installed should determine what entries are put into the table file, e.g. if a product installs headers, the appropriate "_INC" environment variable should be set.
    UPS table files can be tricky to maintain manually due to their multiple clauses for different qualifier combinations, common sections, etc. This can be mitigated by generating them from configuration and build information to which the build system already has access.
  13. The system must be able to produce a list of the distribution package files (i.e. tarballs) corresponding to a specified group of packages.
  14. The system must be able to run, in parallel, builds of non-related packages. This is so that we can make efficient use of multi-core build machines.
  15. The system must be able to take advantage of parallelism within the build of individual packages, when available (e.g. "make -j <ncores>"). This is so that we can make efficient use of multi-core build machines.