Project

General

Profile

Build a Binary Distribution Manifest

Overview

pullProducts consults a distribution manifest for each pull request.
The manifests for binary distributions are automatically created when a release manager runs buildFW.
Distributions are grouped as bundles. Available bundles are found on http://scisoft.fnal.gov/scisoft/bundles/
Only designated release managers have permissions to perform all the steps described below.

NEVER MAKE A BUILD DISTRIBUTION IN YOUR MAIN PRODUCTS DIRECTORY
ALWAYS USE A WORKING PRODUCTS DIRECTORY

Requirements

Before running buildFW, you must pull and unwind the source code tarballs for each package that will be built.
buildFW also requires a cfg script for each distribution.
These instructions presume that appropriate source code tarballs and repository tags are already in place.
build-framework has instructions for making third party source code tarballs.

Naming conventions

  • If a bundle release version is vx_y_z, the bundle version number is x.y.z
  • Bundle release versions are generally the same as the lead product.
    • The art bundle has the same version as the art product.
    • The mu bundle usually has the same version as the art product.
    • The nu bundle usually has the same version as the nutools product.
    • The larsoft bundle has the same version as the larsoft product.
  • cfg script: <bundle name>-cfg-<bundle version number>
  • source code manifest: <bundle name>-<bundle version number>-source_MANIFEST.txt
  • binary distribution manifest: <bundle name>-<bundle version number>-<ups flavor>-<qualifier list>_MANIFEST.txt
    • dashes are used to separate the qualifiers in the qualifier list

Creating the source code manifest and cfg script

We keep both source code manifest templates and cfg templates under version control.
The basic art templates are in the build-framework redmine project.
Most projects, such as larsoft, keep their templates elsewhere for convenience.
The remainder of these instructions presume that both templates are in build-framework.

  • mkdir <source directory>
  • mkdir <build directory>
  • cd <source directory>
  • git clone ssh:///cvs/projects/build-framework
  • edit product version numbers in CMakeLists.txt
  • cd <build directory>
  • source <product directory>/setup (or setups)
  • setup cmake v3_0_1
    • most releases of cmake work with build-framework
  • cmake <source directory>/build-framework
  • make
  • ls art_externals
  • check the newly created files for your distribution:
    • <bundle name>-cfg-<bundle version number>
    • <bundle name>-<bundle version number>-<ups flavor>-<qualifier list>_MANIFEST.txt
    • It is often useful to compare these against the same files in a previous release.
  • If these are as expected, copy them to SciSoft

Building the binary distribution

Basic steps:

mkdir <build directory>
cd <build directory> 
curl -O http://scisoft.fnal.gov/scisoft/bundles/tools/pullProducts
chmod +x pullProducts
./pullProducts `pwd` source <bundle>-<bundle version>
./buildFW -t -b eN [-s sN] `pwd` [debug|prof] <bundle>-<bundle version>

While it is possible to build a binary distribution on your machine, we strongly recommend using Jenkins.
Most Jenkins builds are configured to create debug and prof binary distributions for an appropriate set of supported platforms.
The art team keeps a set of Jenkins build scripts in artutilscripts.
The LArSoft Jenkins build scripts are in larutils.
The build scripts are edited infrequently.

  • General characteristics of a build script:
    • make build and copyBack subdirectories under the Jenkins WORKSPACE directory
    • pull source code
    • pull binary distributions that already exist
      • For instance, the nu distribution will pull the appropriate art binary distribution.
      • This reduces duplication of effort and build time.
    • run buildFW in the build subdirectory
    • copy the manifest and tarballs from the build subdirectory to the copyBack subdirectory
      • Some scripts also copy the individual build logs, but this is not necessary.
  • General characteristics of a Jenkins job:
    • has a descriptive name
    • takes version and qualifiers as options
    • checks out the redmine repository containing the build script
    • runs the build script
    • defines a direcotory (copyBack) for build artifacts

Run a Jenkins job

  • You need to be authorized with Jenkins via a KCA certificate.
  • Make sure you have a valid KCA certificate in your browser.
  • Go to https://buildmaster.fnal.gov/
    • If the name in the upper right is anonymous, you either have an invalid KCA certificate or you need to be authorized.
    • If you have a valid KCA certificate and your full name does not appear in the upper right corner, send an email to requesting authorization to run jobs.
  • There are a number of views: Art, LArSoft, mu2e, etc. These views collect related build scripts so you can easily find the script you need.
  • select the Jenkins job (click on the name)
  • You should see a "Build with Parameters" option on the left.
  • click on "Build with Parameters"
  • edit the parameters appropriately
  • click the "Build" button at the bottom of the form.
    • The round balls will start to blink and will continue to blink while the build is in progress.
    • If the build is successful, all the round balls will be green when they stop blinking.
    • Individual round balls may be either green or red. Red means the build failed.
  • It is possible to save Jenkins job configurations as external xml files:
    curl -E /tmp/x509up_u1000 https://buildmaster.fnal.gov/job/{job-name}/config.xml >{job-name}.xml

Copy to SciSoft

Once the build has completed successfully, you will need to copy the build artifacts to some local directory and then to SciSoft.

  • copyToSciSoft
    • will only copy files which are not already on SciSoft
    • creates the appropriate directory on SciSoft if necessary, and then copies the file
    • copies via scisoftportal.fnal.gov
    • you need a fermiCloud account to use scisoftportal
    • you also need other permissions enabled

Jenkins jobs

This is a partial list of Jenkins jobs used to build various distributions.

Distribution Jenkins job name build script
art art-release-build artutilscripts/for-jenkins/build-art.sh
ifdh build-ifdh artutilscripts/for-jenkins/build-ifdh.sh
larsoft build-larsoft larutils/buildScripts/build-larsoft.sh
mu build-mu-products artutilscripts/for-jenkins/build-mu.sh
nu nu-release-build artutilscripts/for-jenkins/build-nu.sh