Project

General

Profile

Calibration-tagging

Part I: abs/atten/timing constants

When should all of this be done? Whenever you have modified your calibration constants in some way, and are looking to push these (thoroughly tested and validated) constants through to the official calibration. This process should only be done after consulting with the calibration convenors.

These constants ultimately take the form of a UPS product, and are stored as .csv files on /grid/fermiapp/products/nova/externals/calibcsvs/ . This is different from calibration epochs range tables (below), which are stored in a database as of January 2017. As of version 13 we are deprecating the epoch tables for the attenuation and absolute calibration constants. The run ranges defining each epoch will instead be read directly from the filenames.

Pre-requisites

In order to be able to make a new UPS product firstly you need to have permissions to run as the "novasoft" user. Gavin Davies can grant you these permissions. Secondly you need to be granted permission to {add,del,mod} UPS products. Marc Mengel (FNAL) can grant you these.

CSV files

The calibration product consists of a base directory that contains a README, and folders pointing to the varying calibration tags. One set of folders is provided for timing calibration constants (named tvX) and another set for the attenuation and absolute calibrations (named vX). These names correspond to configurable options in Calibrator. To prepare for tagging you should check this folder for the latest calibration product:

$ls /grid/fermiapp/products/nova/externals/calibcsvs/
current.chain   v08.00          v10.00          v11.03          v12.01          v12.08.version
temp            v08.00.version  v10.00.version  v11.03.version  v12.01.version  v12.09
v01.00          v08.01          v10.01          v11.04          v12.02          v12.09.version
v01.00.version  v08.01.version  v10.01.version  v11.05          v12.03          v12.10
v01.01          v08.02          v11.00          v11.06          v12.04          v12.10.version
v01.01.version  v08.02.version  v11.00.version  v11.07          v12.05          v13.00
v07.00          v09.02          v11.01          v11.08          v12.06          v13.00.version
v07.00.version  v09.02.version  v11.01.version  v11.08.version  v12.06.version
v07.01          v09.03          v11.02          v12.00          v12.07
v07.01.version  v09.03.version  v11.02.version  v12.00.version  v12.08

In this case we see that v13.00 is the most recent release. Copy the /NULL/csv/ sub-folder of this into a clean directory:

$ mkdir myStagingDir
$ cd myStagingDir
$ cp -r /grid/fermiapp/products/nova/externals/calibcsvs/v08.02/NULL/csv/ .

Next create a new folder for your new constants (e.g. v14)

$ mkdir v14
$ cd v14

Then copy your new constants into this file. The naming convention is:

calib_{abs,atten}_{consts,points}.{nd,fd}.{data,mc}.{version}.{period}.csv

where version is the tag version (in this case v9) and period is defined by:

rXXX-rXXX - an enclosed period
rXXX-r - an open ended period
r-rXXX - an open started period.
r-r - all runs ever

This convention is enforced by code within calibrator so be sure to follow it carefully. An example can be seen below:

$  ls /grid/fermiapp/products/nova/externals/calibcsvs/v13.00/NULL/csv/v13/
calib_abs_consts.fd.data.v13.r-r20752.csv         calib_atten_consts.nd.mc.v13.r-r.fb4.csv
calib_abs_consts.fd.data.v13.r20753-r23419.csv    calib_atten_consts.nd.mc.v13.r-r.fb5.csv
calib_abs_consts.fd.data.v13.r23420-r24613.csv    calib_atten_consts.nd.mc.v13.r-r.fb6.csv
calib_abs_consts.fd.data.v13.r24614-r29427.csv    calib_atten_consts.nd.mc.v13.r-r.fb7.csv
calib_abs_consts.fd.data.v13.r29428-r.csv         calib_atten_consts.nd.mc.v13.r-r.fb8.csv
calib_abs_consts.fd.mc.v13.r-r20752.csv           calib_atten_consts.nd.mc.v13.r-r.fb9.csv
calib_abs_consts.fd.mc.v13.r20753-r.csv           calib_atten_consts.tb.mc.v13.r-r.csv
calib_abs_consts.nd.data.v13.r-r11228.csv         calib_atten_points.fd.data.v13.r-r20752.csv
calib_abs_consts.nd.data.v13.r11229-r11628.csv    calib_atten_points.fd.data.v13.r20753-r23419.csv
calib_abs_consts.nd.data.v13.r11632-r11925.csv    calib_atten_points.fd.data.v13.r23420-r24613.csv
calib_abs_consts.nd.data.v13.r11926-r12516.csv    calib_atten_points.fd.data.v13.r24614-r.csv
calib_abs_consts.nd.data.v13.r12517-r12955.csv    calib_atten_points.fd.mc.v13.r-r20752.fb0.csv
calib_abs_consts.nd.data.v13.r12956-r.csv         calib_atten_points.fd.mc.v13.r-r20752.fb1.csv
calib_abs_consts.nd.mc.v13.r-r.csv                calib_atten_points.fd.mc.v13.r-r20752.fb2.csv
calib_abs_consts.tb.mc.v13.r-r.csv                calib_atten_points.fd.mc.v13.r-r20752.fb3.csv
calib_atten_consts.fd.data.v13.r-r20752.csv       calib_atten_points.fd.mc.v13.r-r20752.fb4.csv
calib_atten_consts.fd.data.v13.r20753-r23419.csv  calib_atten_points.fd.mc.v13.r-r20752.fb5.csv
calib_atten_consts.fd.data.v13.r23420-r24613.csv  calib_atten_points.fd.mc.v13.r-r20752.fb6.csv
calib_atten_consts.fd.data.v13.r24614-r.csv       calib_atten_points.fd.mc.v13.r-r20752.fb7.csv
calib_atten_consts.fd.mc.v13.r-r20752.fb0.csv     calib_atten_points.fd.mc.v13.r-r20752.fb8.csv
calib_atten_consts.fd.mc.v13.r-r20752.fb1.csv     calib_atten_points.fd.mc.v13.r20753-r.fb0.csv
calib_atten_consts.fd.mc.v13.r-r20752.fb2.csv     calib_atten_points.fd.mc.v13.r20753-r.fb1.csv
calib_atten_consts.fd.mc.v13.r-r20752.fb3.csv     calib_atten_points.fd.mc.v13.r20753-r.fb2.csv
calib_atten_consts.fd.mc.v13.r-r20752.fb4.csv     calib_atten_points.fd.mc.v13.r20753-r.fb3.csv
calib_atten_consts.fd.mc.v13.r-r20752.fb5.csv     calib_atten_points.fd.mc.v13.r20753-r.fb4.csv
calib_atten_consts.fd.mc.v13.r-r20752.fb6.csv     calib_atten_points.fd.mc.v13.r20753-r.fb5.csv
calib_atten_consts.fd.mc.v13.r-r20752.fb7.csv     calib_atten_points.fd.mc.v13.r20753-r.fb6.csv
calib_atten_consts.fd.mc.v13.r-r20752.fb8.csv     calib_atten_points.fd.mc.v13.r20753-r.fb7.csv
calib_atten_consts.fd.mc.v13.r20753-r.fb0.csv     calib_atten_points.fd.mc.v13.r20753-r.fb8.csv
calib_atten_consts.fd.mc.v13.r20753-r.fb1.csv     calib_atten_points.nd.data.v13.r-r11228.csv
calib_atten_consts.fd.mc.v13.r20753-r.fb2.csv     calib_atten_points.nd.data.v13.r11229-r11628.csv
calib_atten_consts.fd.mc.v13.r20753-r.fb3.csv     calib_atten_points.nd.data.v13.r11632-r11925.csv
calib_atten_consts.fd.mc.v13.r20753-r.fb4.csv     calib_atten_points.nd.data.v13.r11926-r.csv
calib_atten_consts.fd.mc.v13.r20753-r.fb5.csv     calib_atten_points.nd.mc.v13.r-r.csv
calib_atten_consts.fd.mc.v13.r20753-r.fb6.csv     calib_atten_points.nd.mc.v13.r-r.fb0.csv
calib_atten_consts.fd.mc.v13.r20753-r.fb7.csv     calib_atten_points.nd.mc.v13.r-r.fb1.csv
calib_atten_consts.fd.mc.v13.r20753-r.fb8.csv     calib_atten_points.nd.mc.v13.r-r.fb10.csv
calib_atten_consts.nd.data.v13.r-r11228.csv       calib_atten_points.nd.mc.v13.r-r.fb11.csv
calib_atten_consts.nd.data.v13.r11229-r11628.csv  calib_atten_points.nd.mc.v13.r-r.fb2.csv
calib_atten_consts.nd.data.v13.r11632-r11925.csv  calib_atten_points.nd.mc.v13.r-r.fb3.csv
calib_atten_consts.nd.data.v13.r11926-r.csv       calib_atten_points.nd.mc.v13.r-r.fb4.csv
calib_atten_consts.nd.mc.v13.r-r.csv              calib_atten_points.nd.mc.v13.r-r.fb5.csv
calib_atten_consts.nd.mc.v13.r-r.fb0.csv          calib_atten_points.nd.mc.v13.r-r.fb6.csv
calib_atten_consts.nd.mc.v13.r-r.fb1.csv          calib_atten_points.nd.mc.v13.r-r.fb7.csv
calib_atten_consts.nd.mc.v13.r-r.fb10.csv         calib_atten_points.nd.mc.v13.r-r.fb8.csv
calib_atten_consts.nd.mc.v13.r-r.fb11.csv         calib_atten_points.nd.mc.v13.r-r.fb9.csv
calib_atten_consts.nd.mc.v13.r-r.fb2.csv          calib_atten_points.tb.mc.v13.r-r.csv
calib_atten_consts.nd.mc.v13.r-r.fb3.csv

Once you've copied your files in place and tested them thoroughly you're ready to tag. This is done via a novasoft script in SRT_NOVA/scripts/make_calib_csvs_product.sh.

Make sure that you're logged in as novasoft but do not run setup_nova.

You can then run:

$ /cvmfs/nova-development.opensciencegrid.org/novasoft//releases/development/SRT_NOVA/scripts/make_calib_csvs_product.sh <input folder> <UPS product name>

Important: do not use relative path (e.g. ".") but the absolute (starting "/"), the script will make the wrong product otherwise.

You should see no errors, although there will be things that look like errors and are to do with move_archive_file, apparently these are harmless. An example of this is

$ /cvmfs/nova-development.opensciencegrid.org/novasoft//releases/development/SRT_NOVA/scripts/make_calib_csvs_product.sh /nova/app/users/rjn/calibration/staging/version13/csv/ v13.00

Location of calibration CSV files: /nova/app/users/rjn/calibration/staging/version13/csv/
Name of product: calibcsvs
Version: v13.00

Product will be created in /grid/fermiapp/products/nova/externals/calibcsvs
Copying CSV files from /nova/app/users/rjn/calibration/staging/version13/csv/ to /grid/fermiapp/products/nova/externals/calibcsvs/v13.00/NULL/csv/
Creating table file..
Declaring product calibcsvs with version v13.00 to UPS.
Adding product to UPD..
Making a tarball to avoid /tmp file size limitations: /grid/fermiapp/products/nova/externals/calibcsvs/temp/calibcsvs.v13.00.upd.addproduct.tar.gz
Error output of move_archive_file: Authenticated rjn@FNAL.GOV
    Account updadmin: authorization for rjn@FNAL.GOV for execution of
                   /usr/krb5/k5arc/scripts/upd successful
    Changing uid to updadmin (100)
error output of move_ups_dir: Authenticated rjn@FNAL.GOV
    Account updadmin: authorization for rjn@FNAL.GOV for execution of
                   /usr/krb5/k5arc/scripts/upd successful
    Changing uid to updadmin (100)
upd addproduct succeeded.
Removing the tarball and directory: /grid/fermiapp/products/nova/externals/calibcsvs/temp/calibcsvs.v13.00.upd.addproduct.tar.gz
rm: descend into directory `/grid/fermiapp/products/nova/externals/calibcsvs/temp'? y
rm: remove regular file `/grid/fermiapp/products/nova/externals/calibcsvs/temp/calibcsvs.v13.00.upd.addproduct.tar.gz'? y
rm: remove directory `/grid/fermiapp/products/nova/externals/calibcsvs/temp'? y
All done!

You should then check that your calibration product shows up via:

$ ls /grid/fermiapp/products/nova/externals/calibcsvs/
current.chain   v08.00.version  v10.00.version  v11.03.version  v12.01.version  v12.09
v01.00          v08.01          v10.01          v11.04          v12.02          v12.09.version
v01.00.version  v08.01.version  v10.01.version  v11.05          v12.03          v12.10
v01.01          v08.02          v11.00          v11.06          v12.04          v12.10.version
v01.01.version  v08.02.version  v11.00.version  v11.07          v12.05          v13.00
v07.00          v09.02          v11.01          v11.08          v12.06          v13.00.version
v07.00.version  v09.02.version  v11.01.version  v11.08.version  v12.06.version
v07.01          v09.03          v11.02          v12.00          v12.07
v07.01.version  v09.03.version  v11.02.version  v12.00.version  v12.08
v08.00          v10.00          v11.03          v12.01          v12.08.version

You can also check what is setup in a version of novasoft via:

$ setup_nova; setup calibcsvs v08.00; ls -ltrR $CALIBCSVS_DIR/

Removing broken products

This can be done with:

ups undeclare -y "calibcsvs" "v09.04" -f "NULL" 

Updating novasoft to point at your new version of Calibration constants

In files:

novasoft/setup/nova-offline-ups-externals-development
novasoft/setup/nova-offline-ups-externals-development-prof

and

novasoft/SRT_NOVA/scripts/install_scripts/install_nova_externals_development

and

novasoft/ups/product_deps

simply search for the string 'calibcsvs'. You will see that it is followed by a version, like v10.00. Change this to the version that you just tagged, and commit these changes.

If you have made any changes to the packages Calibration, CalibrationDataProducts, CalibrationUtils, or Calibrator, make certain to commit those as well!

Also, please remind the production group to publish the new version of calibration constants in CVMFS. This is not something that you will necessarily have permissions to do yourself, but you are asked to please ask those that do (the production group) to publish in CVMFS.

Merging with a branch

You may be asked by production to commit the changes described above to a branch, rather than development. See this page or ask your production convener if you need instructions on how to do this.

Telling novasoft and Calibrator to use the new constants

Finally, make certain that you have also told the Calibrator to use these tags in Calibrator/art/Calibrator.fcl, namely by updating the lines:


Tag: "v13”

TimingTag: "tv4"

After you have completed these steps, inform the production group so that they can begin producing calibrated files with this new tag.

Merging with a branch

You may be asked by production to commit the changes described above to a branch, rather than development. See this page or ask your production convener if you need instructions on how to do this.