Project

General

Profile

Creating UPS products

If you have a package that has been approved to become a UPS product then there a few steps that need to be made in order to declare it to the UPS database.
Several different types of UPS products can be created. We make UPS products of the novasoft, version-controlled packages NovaGridUtils and novaproduction as well as products
for the PID libraries, custom externals etc. The method varies in each case.

Creating NGU/novaproduction UPS products

This is done fairly regularly so there's a script to automate this process called make_nova_ups_product.sh
This must be executed as novasoft and requires two arguments:

  1. The product name of UPS product to be created
  2. The version number of UPS product to be created

Example:

% $SRT_PUBLIC_CONTEXT/SRT_NOVA/scripts/make_nova_ups_product.sh NovaGridUtils v01.50

This declares the new product to UPS and adds it for possible further distribution to UPD. It is, at this point, available on the interactive gpvm's (installed in /grid/fermiapp/products/nova/externals and available for setup.
You can confirm this by seing your new product and version via:
% ups list -aK+ NovaGridUtils

which lists all available versions in ups.

Creating PID library UPS products

In a similar fashion to NGU and/or novaproduction there's a script to automate creation of the PID library UPS products, namely make_pidlib_product.sh
Once again, need to be user novasoft to run this script and this one requires three arguments:

  1. The location of PID library files
  2. PID package name
  3. Version number of UPS product to be created

Example:

% $SRT_PUBLIC_CONTEXT/SRT_NOVA/scripts/make_pidlib_product.sh /path/to/pidlibs/ lem v01.20

Encourage users to create a README file and keep pid library filenames sensible. This script doesn't keep the README along for the ride (we should change that).
In the interim, I've been creating the products manually to retain the README but perhaps we should encourage the README to be committed and live with the package the
libraries accompany.

Look packages in /nova/data/pidlibs/products/ and $EXTERNALS (/grid/fermiapp/products/nova/externals).
Don't forget to modify:
nova-offline-ups-externals-development(-prof)
install_nova_externals_development
and push upd to CVMFS

Creating Calibration calibcsvs UPS product

In all likelihood, this will be handled by a member of the Calibration group.
Nevertheless, in a similar fashion to NGU and/or novaproduction, there's a script to automate creation of the calibcsvs UPS product, namely make_calib_csvs_product.sh
Once again, need to be user novasoft to run this script and this one requires two argument:

  1. The location of the calibration CSV files (/nova/ana/calibration/csvs/?)"
  2. Version number of UPS product to be created"

Example:

% $SRT_PUBLIC_CONTEXT/SRT_NOVA/scripts/make_calib_csvs_product.sh /path/to/csvs/ v08.02

For more detailed instructions see Calibration-tagging.

Creating GiBUU libs UPS product

Should you need to make a new UPS product of GiBUU libraries, there's a script for that; namely make_gibuu_libs_product.sh
Once again, need to be user novasoft to run this script and this one requires two arguments:

  1. The location of the GiBUU libraries
  2. Version number of UPS product to be created"

Example:

% $SRT_PUBLIC_CONTEXT/SRT_NOVA/scripts/make_gibuu_libs_product.sh /path/to/gibuu_libs/ v2016

Pushing UPS products to CVMFS

Once you've created said ups products they can be setup on the gpvms but not CVMFS just yet. You need to follow the next steps to push them to cvmfs:
Log on to cvmfs:

% nova_cvmfs_check
% nova_cvmfs_transaction
% source /cvmfs/nova.opensciencegrid.org/externals/setup
% setup upd
% upd install <product_name> <product_version>
% nova_cvmfs_publish

When all that is said and done, and you're ready to make these setup as default then one needs to make commits to three places:

  • setup/nova-offline-ups-externals-development
  • setup/nova-offline-ups-externals-development-prof
  • SRT_NOVA/scripts/install_scripts/install_nova_externals_development
  • ups/product_deps # for CMake users

Creating custom products

We follow the example of declaring the external package cppcheck as a UPS product. This package is free, open-source and does not require a license so is approved.

Locate/Download package

Locate the correct version and its corresponding zip/tar file and download to a test area, i.e. your home directory.

% wget http://sourceforge.net/projects/cppcheck/files/cppcheck/1.63/cppcheck-1.63.tar.bz2;
% tar xjvf cppcheck-1.63.tar.bz2;
% cd cppcheck-1.63;

This grabs v1_63 of cppcheck and unwinds it - the most recent version at the time of writing.

Compile package

Follow installation instructions from the package's README file (in this case readme.txt)

% g++ -o cppcheck -Ilib -Iexternals/tinyxml cli/*.cpp lib/*.cpp externals/tinyxml/tinyxml2.cpp

I recommend you setup novasoft first so you get the latest version of gcc compiler being used in NOvA.

Tidy up directory for UPS

More detailed help is here: https://cdcvs.fnal.gov/redmine/projects/ups/wiki/Develop#7566
cppcheck package-specific details are below.

Rename cppcheck-1.63 to cppcheck and make a directory for v1_63 so subsequent versions can live under the same roof but their own version directory.
Make bin/ and ups/ directories. Move executable to bin/ directory.

% cd ..; mkdir -p cppcheck/v1_63/NULL/bin cppcheck/v1_63/ups;
% mv cppcheck-1.63/* cppcheck/v1_63/NULL;
% cd cppcheck/v1_63/NULL;
% mv cppcheck bin/ 

Create a .table file

Create a cppcheck.table file in the ups/ directory and fill it with the following contents:

FILE=TABLE 
PRODUCT=cppcheck
VERSION=v1_63
# 
#--------------------- 
FLAVOR = ANY 
QUALIFIERS = "" 

ACTION=SETUP 
   pathPrepend(PATH, ${UPS_PROD_DIR}/bin )
   EnvSet(CPPCHECK_VERSION, ${UPS_PROD_VERSION} )
   EnvSet(CPPCHECK_LIB_PATH, ${UPS_PROD_DIR}/lib )
   setupEnv() 

Declare package to UPS DB

Now we can move our package to /grid/fermiapp/products/nova/externals area. Ask someone with user "nusoft" permissions to do the next steps.
They will then declare the package to the UPS Database.

% ssh novasoft@novagpvmXX.fnal.gov; # wherever you like on the VMs
% cp -r /path/to/package/cppcheck/ /grid/fermiapp/products/nova/externals;
% source /grid/fermiapp/products/nova/externals/setup;
% cd $PRODUCTS;
% ups declare cppcheck v1_63 -f NULL -m cppcheck.table -z $PRODUCTS -r cppcheck/v1_58;

The environment variable $PRODUCTS points to /nusoft/app/externals. We declare this particular package with NULL flavour because it is unflavoured.

Check package declaration

One can now check that the package was successfully declared.

% ups list -alz $PRODUCTS cppcheck;

DATABASE=/nusoft/app/externals 
    Product=cppcheck    Version=v1_63    Flavor=NULL
        Qualifiers=""    Chain="" 
        Declared="2013-03-14 00.52.40 GMT" 
        Declarer="nusoft" 
        Modified="2013-03-14 00.52.40 GMT" 
        Modifier="nusoft" 
        Home=/nusoft/app/externals/cppcheck/v1_63
        No Compile Directive
        Authorized, Nodes=*
        UPS_Dir="ups" 
        Table_Dir="" 
        Table_File="cppcheck.table" 
        Archive_File="" 
        Description="" 
        Action=SETUP
            pathPrepend(PATH, ${UPS_PROD_DIR}/bin, :)
            setupEnv()

Setup your package

You can go ahead and setup the package and start using it.

% setup cppcheck v1_63;
% cppcheck -h;

Display the help to show it was a success! The setup command should be added to $SRT_DIST/setup/nova-offline-ups-externals-development and $SRT_DIST/setup/nova-offline-ups-externals-development-prof for all users to see this package after setting up novasoft. Also add package installation to $SRT_DIST/scripts/install_nova_externals_development.

Create tarball

Experts: Now create tarball for external users.

% ssh novasoft@novagpvmXX.fnal.gov
% cd /grid/fermiapp/products/nova/externals
% tar -czvf cppcheck/cppcheck-v1_63.tar.gz cppcheck/v1_63 cppcheck/v1_63.version/
</pre

	

and update the Installing a local copy of NOvASoft and the external products page if appropriate.