Project

General

Profile

UPS and CVMFS Installations for Development Work

These are instructions of setting up to do interactive development work and using the build system. Interactive work can be done on the VM geniegpvm01.fnal.gov


Setting up external products using UPS

Common pre-built external products such as gcc, ROOT, LHAPDF, log4cpp, etc are built by Lynn Garren for use by the experiments. To avoid the need to build these on our own we use a similar procedure for downloading and installing those pre-built versions.

To bootstrap UPS one does one or the other of:

  source /cvmfs/fermilab.opensciencegrid.org/products/genie/externals/setup
  source /grid/fermiapp/products/genie/externals/setup
depending on whether one want to access the version on CVMFS or /grid/fermiapp.

A minimal setup of externals necessary for GENIE development could be:

  setup root v5_34_25a -q debug:e7:nu
    # the root setup in turn ensures a compatible gcc, gsl, pythia6, and libxml2
  setup lhapdf v5_9_1b -q debug:e7
  setup log4cpp v1_1_1b -q debug:e7

If you want to setup a pre-built version of GENIE with cross section splines and have it decay charm hadrons and taus:

  setup genie v2_8_6a -q debug:e7
  setup genie_xsec v2_8_6 -q default
  setup genie_phyopt v2_8_6 -q dkcharmtau   # or dkcharm

Listing available products and dependencies

To list the available version of a particular product use:

   ups list -aK+ <prd-name>

leaving out the product name will list all products.

To see the dependency tree use:

   ups depend genie [-B] v2_8_6a -q debug:e7

where the option -B will show repeat sub-dependencies, while without it only the first instance is shown. The output (sans -B) looks like:
genie v2_8_6a -f Linux64bit+2.6-2.12 -z /grid/fermiapp/products/genie/externals -q debug:e7
|__root v5_34_25a -f Linux64bit+2.6-2.12 -z /grid/fermiapp/products/genie/externals -q debug:e7:nu
|  |__clhep v2_2_0_5 -f Linux64bit+2.6-2.12 -z /grid/fermiapp/products/genie/externals -q debug:e7
|  |  |__gcc v4_9_2 -f Linux64bit+2.6-2.12 -z /grid/fermiapp/products/genie/externals
|  |__fftw v3_3_4 -f Linux64bit+2.6-2.12 -z /grid/fermiapp/products/genie/externals -q debug
|  |__gsl v1_16 -f Linux64bit+2.6-2.12 -z /grid/fermiapp/products/genie/externals -q debug
|  |__pythia v6_4_28c -f Linux64bit+2.6-2.12 -z /grid/fermiapp/products/genie/externals -q debug:gcc492
|  |__postgresql v9_3_6 -f Linux64bit+2.6-2.12 -z /grid/fermiapp/products/genie/externals -q p279
|  |  |__python v2_7_9 -f Linux64bit+2.6-2.12 -z /grid/fermiapp/products/genie/externals
|  |     |__sqlite v3_08_08_02 -f Linux64bit+2.6-2.12 -z /grid/fermiapp/products/genie/externals
|  |__mysql_client v5_5_42 -f Linux64bit+2.6-2.12 -z /grid/fermiapp/products/genie/externals -q e7
|  |__xrootd v3_3_4c -f Linux64bit+2.6-2.12 -z /grid/fermiapp/products/genie/externals -q debug:e7
|  |__libxml2 v2_9_2 -f Linux64bit+2.6-2.12 -z /grid/fermiapp/products/genie/externals -q debug
|__lhapdf v5_9_1b -f Linux64bit+2.6-2.12 -z /grid/fermiapp/products/genie/externals -q debug:e7
|  |__pdfsets v5_9_1b -f NULL -z /grid/fermiapp/products/genie/externals
|__log4cpp v1_1_1b -f Linux64bit+2.6-2.12 -z /grid/fermiapp/products/genie/externals -q debug:e7

These might help with additional options (man pages might not be accessible):

   ups list -?
   ups depend -?
   man ups_list
   man ups_depend

Problems solving

If you see a situation like:
$ source /grid/fermiapp/products/genie/externals/setup
/usr/bin/which: no ups in (/afs/fnal.gov/files/home/room2/xyzzy/linux/bin:/usr/krb5/bin:/usr/kerberos/bin:/opt/condor/bin:/opt/condor/sbin:/usr/local/bin:/bin:/usr/bin)
ERROR: did not find ups binary

this is probably because the version of UPS is not installed for the particular flavor of the OS.
$ cat /etc/redhat-release 
Scientific Linux SLF release 5.5 (Lederman)

which is incompatible with an installation of only the SLF6 version.
OS release UPS "flavor" uname -r /lib/libc*
SLF5 Linux[64bit]+2.6-2.5 2.6.18-402.el5 GNU C Library stable release version 2.5
SLF6 Linux[64bit]+2.6-2.12 2.6.32-504.8.1.el6.x86_64 GNU C Library stable release version 2.12

Using the env variables defined by UPS

When configuring to build GENIE use:

   --with-pythia6-lib=${PYTHIA_FQ_DIR}/lib
   --with-lhapdf-inc=${LHAPDF_FQ_DIR}/include
   --with-lhapdf-lib=${LHAPDF_FQ_DIR}/lib
   --with-libxml2-inc=${LIBXML2_FQ_DIR}/include/libxml2
   --with-libxml2-lib=${LIBXML2_FQ_DIR}/lib
   --with-log4cpp-inc=${LOG4CPP_FQ_DIR}/include
   --with-log4cpp-lib=${LOG4CPP_FQ_DIR}/lib

Installation of new pre-built versions of externals (bundles) into UPS

ssh geniegpvm01.fnal.gov

# make what gets written group writable since we're not using a shared account
umask 0002

# convenient shorthands
export EXTERNALS=/grid/fermiapp/products/genie/externals
export UPS_WORK_AREA=/genie/app/ups_work_area

# ensure the directories exist
if [ ! -d ${EXTERNALS} ]; then mkdir -p ${EXTERNALS} ; fi
if [ ! -d ${UPS_WORK_AREA} ]; then mkdir -p ${UPS_WORK_AREA} ; fi

# work area -- tar files end up here
cd ${UPS_WORK_AREA}

# fetch the script, make it executable
curl -O http://scisoft.fnal.gov/scisoft/bundles/tools/pullProducts
chmod +x pullProducts

# help keep track of what was done by keeping the log
NOW=`date "+%Y%m%d-%H%M%S"`

# define what we're going to pull
# most nodes at FNAL are now SLF6
# for now WHAT should be "nubase" (includes a version of genie that the experiments use, but that's okay) 
# and the choices are version/qualifier combinations
# and yes, installed version have vX_Y_Z while tarballs and manifests use X.Y.Z notation (ugh)
# QUAL2 can be "debug" or "prof", you might want to install both
#
OS=slf6
WHAT=nubase
VERSION=v1_10_01
QUAL1=e7
QUAL2=debug

# do the actual pull / unpack
nohup ./pullProducts ${EXTERNALS} $OS ${WHAT}-${VERSION} ${QUAL1} ${QUAL2} > zzz_pull.$OS.${WHAT}-${VERSION}_${QUAL1}_${QUAL2}.${NOW} 2>&1 &

# if you were successful then clean things out
rm *.tar.gz2

# this should leave some files that look like:
# MANIFEST - list of products installed
nubase-1.10.01-Linux64bit+2.6-2.12-e7-debug_MANIFEST.txt
nubase-1.10.01-Linux64bit+2.6-2.12-e7-prof_MANIFEST.txt
# dated log files
zzz_pull.slf6.nubase-v1_10_01_e7_debug.20150430-140900
zzz_pull.slf6.nubase-v1_10_01_e7_prof.20150504-173200

What's available?

  curl -O http://scisoft.fnal.gov/scisoft/bundles/tools/getManifestList
  chmod +x getManifestList
  ./getManifestList
This script will list bundles, versions for a specific bundle, and available manifests for a specific bundle and version.

Doing bits by hand

Sometimes one needs versions of packages not in the nubase bundle, these can be fetched and unpacked with a procedure along the lines of:

ssh geniegpvm01.fnal.gov

# make what gets written group writable since we're not using a shared account
umask 0002

# convenient shorthands
export EXTERNALS=/grid/fermiapp/products/genie/externals
export UPS_WORK_AREA=/genie/app/ups_work_area

# ensure the directories exist
if [ ! -d ${EXTERNALS} ]; then mkdir -p ${EXTERNALS} ; fi
if [ ! -d ${UPS_WORK_AREA} ]; then mkdir -p ${UPS_WORK_AREA} ; fi

# work area -- tar files end up here
cd ${UPS_WORK_AREA}

curl -O https://scisoft.fnal.gov/scisoft/packages/gdb/v7_9/gdb-7.9-slf6-x86_64.tar.bz2
cd $EXTERNALS
tar xjf ${UPS_WORK_AREA}//gdb-7.9-slf6-x86_64.tar.bz2

CVMFS

The allocated CVMFS areas for GENIE are:

   /cvmfs/fermilab.opensciencegrid.org/products/genie/
   /cvmfs/fermilab.opensciencegrid.org/genie/

These are mapped from:

   /grid/fermiapp/products/genie/
   /grid/fermiapp/genie/

As of 2015-05-04 these source areas should get indexed nightly at 4am, the logs from which can be found at:

  /grid/fermiapp/cvmfsfermilab/logs/sync-cvmfs-fermilab.log

The work area

   /genie/app/
is not indexed to CVMFS at all and should be used for staging and personal development.

Triggering update & Permissions issues

Per Dave Dykstra 2015-05-05: Assuming the directory /grid/fermiapp/cvmfsfermilab/triggers/genie exists one can trigger additional updates when you need them, from the genie group account (geniepro), by doing:

ssh geniepro@geniegpvm01.fnal.gov
nohup /grid/fermiapp/cvmfsfermilab/bin/sync-cvmfs genie > /genie/app/$USER/sync.out 2>/genie/app/$USER/sync.err &

Check the output files for "rsync:" messages which indicate a problem. All files must be world readable and all directories readable and executable. These can be fixed with:

   find . -type d ! -perm -5   | xargs chmod a+rx
   find . -type d ! -perm -a+r | xargs chmod a+r

Ensuring that CVMFS is up-to-date on a node

   /cvmfs/grid.cern.ch/util/cvmfs-uptodate  /cvmfs/fermilab.opensciencegrid.org/products/genie
   /cvmfs/grid.cern.ch/util/cvmfs-uptodate  /cvmfs/fermilab.opensciencegrid.org/genie

Dave Dykstra says one will need to do this until at least the end of 2015...


Useful Links

Bundles are located on:

http://scisoft.fnal.gov/

https://cdcvs.fnal.gov/redmine/projects/cet-is-public/wiki/BuildBinaryDistributionManifest