Project

General

Profile

A step-by-step guide to building artdaq with Jenkins

JCF, Jun-7-2017: these instructions are now largely obsolete, thanks to sands shifting underfoot

Overview

These instructions are intended to allow members of the RSI group the ability to perform a Jenkins build by following a cookbook series of steps. To the greatest extent possible, an attempt has been made to protect the reader from having to "fill in the gaps" or rely on oral tradition in order to accomplish a Jenkins build.

The basic idea is that you build your package -- e.g., artdaq, lbne-raw-data, etc. -- using the Jenkins interface webpage. After providing it with the names of a couple of scripts which you will first need to produce yourself off of templates, you hit a button which results in a build, and if the scripts you provided don't have any errors, over a period of 15 or 20 minutes a binary of your package is built, which you then upload to SciSoft.

Permissions that you'll want

Creating the two files Jenkins needs: the source code manifest, and the build config (buildcfg) file

Note that the instructions in this section supersede the instructions in the (collapsed) section farther down this wiki "Old instructions prior to the creation of the script"; this is because many of the steps in the older instructions are now scripted to be performed automatically

First, check out the latest version of artdaq-utilities via git clone ssh://p-artdaq-utilities@cdcvs.fnal.gov/cvs/projects/artdaq-utilities. The script we'll be running is called "for_jenkins/create_all_jenkins_builds.sh" with the path given relative to the artdaq-utilities directory just checked out. We'll want to run it out of a new directory that's not a subdirectory of the package itself, so, assuming we checked out artdaq-utilities into <basedir> and that we're working out of the newly created <newdirectory>, the command is:

<basedir>/artdaq-utilities/for_jenkins/create_all_jenkins_builds.sh <packagename> <packageversion> (optional name of file containing list of qualifiers)

The first two arguments are simply the package name and its version; the third argument is optional, and is the name of a file containing a list of qualifiers sets which you want factored into the creation of the source manifest. So., e.g., if the contents of a file called <newdirectory>/qualifiers_we_care_about.txt are as follows:
e7:s11:eth
e7:s15:eth
e7:s11:ib
e7:s15:ib

Then if we run
<basedir>/artdaq-utilities/for_jenkins/create_all_jenkins_builds.sh artdaq v1_12_12 qualifiers_we_care_about.txt

we'll get the source manifest for artdaq v1_12_12 both for art v1_15_02 and v1_14_02, as well as ethernet and infiniband. On the other hand, if we run
<basedir>/artdaq-utilities/for_jenkins/create_all_jenkins_builds.sh artdaq v1_12_12 

then the source manifest will be created for all qualifier combinations provided in the ups/product_deps file for artdaq v1_12_12.

In general, it takes a few minutes to run the script; the lion's share of the time is spend git clone-ing both the target package as well as the packages it depends on, in order that their ups/product_deps files can be studied by the scripts.

n.b. If we're moving to a new major version of art (e.g., from v1_15_02 to v1_16_02), it will probably be necessary to update the calculation of the build_framework_branch variable in artdaq-utilities/for_jenkins/create_jenkins_build.sh . So for the specific move just described, I had to make the edit from

if [[ "$major_art_version" == "v1_15" ]]; then
    build_framework_branch="for_art_v1_15" 
// rest of if - elif -else block omitted here...

to

if [[ "$major_art_version" == "v1_16" ]]; then
    build_framework_branch="master" 
elif [[ "$major_art_version" == "v1_15" ]]; then
    build_framework_branch="for_art_v1_15" 
// rest of if - elif -else block omitted here...

Sometimes the scripts will be unable to handle a subset of the qualifier combinations provided to it. Reasons for this may include the following:

-Some qualifier combinations may have been accidentally included in a ups/product_deps file. If, for example, a package depends on artdaq-core vAA_BB_CC and contains an s<n> qualifier where <n> represents an art version which artdaq-core vAA_BB_CC doesn't support, create_all_jenkins_builds.sh will fail for that particular qualifier combination and move on to the next combination

-Another failure mode is if a qualifier combination results in a dependence on art v1_12_XX or earlier ; the script is not equipped to handle this

If there are any qualifier combinations which create_all_jenkins_builds.sh was unable to process, they're listed at the end of the running of the script. In general, there's no practical effect to this failure other than that the package versions corresponding to that qualifier combination are not guaranteed to be in the source manifest produced. Other information is listed at the end of running as well; an example of what to expect at the end of executing the script is the following (contents explained below):

Edits (if any) performed on artdaq-utilities; please commit them so Jenkins can see them
diff --git a/for_jenkins/build-artdaq.sh b/for_jenkins/build-artdaq.sh
index ecf2183..7fce717 100755
--- a/for_jenkins/build-artdaq.sh
+++ b/for_jenkins/build-artdaq.sh
@@ -21,6 +21,27 @@ qual_set="${QUAL}" 
 build_type=${BUILDTYPE}

 case ${qual_set} in
+s8:e6)
+       basequal=e6
+       squal=s8
+       artver=v1_13_02
+       nutoolsver=v1_10_01
+       ;;
+ 
+s11:e6)
+       basequal=e6
+       squal=s11
+       artver=v1_14_02
+       nutoolsver=v1_11_01
+       ;;
+ 
+s15:e6)
+       basequal=e6
+       squal=s15
+       artver=v1_15_02
+       nutoolsver=v1_14_02
+       ;;
+ 
   s15:e7)
      basequal=e7
      squal=s15

Build configuration file is artdaq-buildcfg-1.12.12 - pls. check to ensure it supports all relevant art versions
Source manifest file is artdaq-1.12.12-source_MANIFEST.txt

Successfully processed qualifier combinations (if any):
e7:s15:eth
e7:s11:eth
e7:s8:eth
e6:s15:eth
e6:s11:eth
e6:s8:eth
g360:e7:s15:eth
g360:e7:s11:eth
g360:e7:s8:eth
g360:e6:s15:eth
g360:e6:s11:eth
g360:e6:s8:eth
g317:e7:s11:eth
g317:e7:s8:eth
g317:e6:s15:eth
g317:e6:s11:eth
g317:e6:s8:eth

Unsuccessfully processed qualifier combinations (if any):
g317:e7:s15:eth

First, a diff is performed with the HEAD of artdaq-utilities for the file for_jenkins/build-<package>.sh (here, for_jenkins/build-artdaq.sh). A case-switch statement for different qualifier combinations near the top of the script will set a few variables; if a given qualifier combination isn't in the for_jenkins/build-<package>.sh file, it's automatically edited. In the example above, you can see that e6:s8, e6:s11 and e6:s15 had not originally been included. Please note you'll want to push these edits to the central repo so Jenkins can use them when you're on its website performing the build.

Next, the name of the source manifest and build configuration files are given. The "Build configuration file is artdaq-buildcfg-1.12.12 - pls. check to ensure it supports all relevant art versions" statement reflects that of this writing (9/10/15) there's no guarantee that the build configuration file will account for all qualifiers you specified. In fact, it's definitely recommended that you make sure the file in the format "<packagename>-buildcfg-<packageversion>" looks more like "<packagename>-buildcfg-<packageversion>.<latest qualifiers>" than "<packagename>-buildcfg-<packageversion>.<olderqualifiers>"; if this isn't the case, you're better off just copying "<packagename>-buildcfg-<packageversion>.<latest qualifiers>" into "<packagename>-buildcfg-<packageversion>" . The source manifest should be fine, though it never hurts to check.

Finally, the successfully processed qualifier combinations and the unsuccessfully processed qualifier combinations are listed.

Please note that the output of create_all_jenkins_builds.sh are saved in two scripts: one containing the complete output, and the other containing the output to stderr. They have names such as the following:

-rw-r--r--. 1 jcfree g163   3060 Sep 10 15:57 create_all_jenkins_builds.sh_Thu_Sep_10_15:57:22_stderr.script
-rw-r--r--. 1 jcfree g163 159825 Sep 10 15:57 create_all_jenkins_builds.sh_Thu_Sep_10_15:57:22.script

Once you have these two files (and after you've double checked that the lbne_raw_data-buildcfg-X.YY.ZZ is to your liking), you'll want to copy them to SciSoft. To crib some of Lynn Garren's notes at https://cdcvs.fnal.gov/redmine/projects/cet-is-public/wiki/BuildBinaryDistributionManifest , you'd do the following; note you'll want to adjust your exact syntax to reflect the proper package and version (lbne-raw-data v1_03_04 is provided in the example below):

curl -O http://scisoft.fnal.gov/scisoft/bundles/tools/copyToSciSoft
chmod +x copyToSciSoft
copyToSciSoft lbne_raw_data-1.03.04-source_MANIFEST.txt
copyToSciSoft lbne_raw_data-buildcfg-1.03.04 

Go to the Jenkins build website, and get it to build your package

This website is at https://buildmaster.fnal.gov ; as mentioned before, your browser will need a KCA certificate in order for this to work. [STILL NEED TO UNDERSTAND HOW TO NAVIGATE THE WEBSITE BEFORE ADDING DETAILS HERE]

Copy the results to SciSoft

After a successful Jenkins build, simply follow Lynn's instructions under "Copy to SciSoft" at https://cdcvs.fnal.gov/redmine/projects/cet-is-public/wiki/BuildBinaryDistributionManifest#Copy-to-SciSoft

Notes from John to himself

-After cutting a release using gitflow, you still need to execute "git push --tags"
-Remember to update the relevant file in the for_jenkins/ subdirectory of artdaq-utilities
-For buildmaster, you need a certificate from https://cilogon.org/