- Table of contents
- Preamble
- NOvA Software in CVMFS
- Checking CVMFS
Preamble¶
This site is mainly intended as a reference manual for super users (i.e. current and past NOvA offline release managers) to document any details necessary for maintaining, pubishing and distributing software via CVMFS.
NOvA Software in CVMFS¶
The full nova-art/nusoft distribution has been made fully relocatable so that it can be hosted off of a number of different platforms that allow for code distribution. This allows the nova software to be made available offsite to remote computing clusters and to individual uses who want to easily run NOvA jobs from their laptop or desktop computers (without the hassle of maintaining a base release).
Currently (Feb 2016) we have distributions of the NOvA software installed on a Cern Virtual Machine File System (CVMFS) servers which distributes the code to various sites. CVMFS is the same technology that Atlas and CMS use to distribute their software distributions because it features a just-in-time file delivery scheme with local caching (meaning copy only those files that you actually use to your local machine when you use them). This makes the system especially attractive when using the NOvA software system, since the full base release + external packages is over 150 GB, while the actual run time foot print of most jobs is only 1.5-2 GB.
Our CVMFS is hosted and maintained at Fermilab. We formerly used the Oasis CVMFS server (not hosted or maintained at Fermilab), but stopped using it in the summer of 2015. We made the switch to improve turnaround times for handling problems, and also speed up the process of publishing our software to CVMFS.
The following sections detail what has been done to setup the NOvA software on the CVMFS servers and what needs to be done to use it at different remote sites.
Overview¶
Overall, the process for publish to CVFMS is as follows:
- Login to the CVMFS server
- Put the repository into a transaction
- copy the software to be published to the destination (for example through upd)
- Optionally verify that the copy was successful (but highly recommended)
- publish updated repository
Login¶
Access to super users is controlled via a
~/.k5loginfile. Just ask for your principle to be added by someone that already has access.
Use this command to log in:
ssh -l cvmfsnova oasiscfs.fnal.gov
Workflow (modifying / adding / publishing)¶
Once you have logged on to the FNAL CVMFS server you MUST put the repository in to transaction before making any modifications / additions / deletes to the repository. If you don't you can put the server into a non-trivial state, and may need to remount the repository (there are some links later to help deal with such situations that can occur).
Transaction start / check¶
Use these commands to check if a transaction has been started / start a transaction:
nova_cvmfs_check nova_cvmfs_transaction
NB. These are just wrappers for this command:
cvmfs_server transaction nova.opensciencegrid.org
Software Locations¶
Common locations:
/cvmfs/nova.opensciencegrid.org/ #base directory /cvmfs/nova.opensciencegrid.org/novasoft/slf5/novasoft/ #SLF5 /cvmfs/nova.opensciencegrid.org/novasoft/slf6/novasoft/ #SLF6 /cvmfs/nova.opensciencegrid.org/externals #external products
Copying release from Jenkins¶
nova_cvmfs_copy_release_from_jenkins $release $os $buildnum nova_cvmfs_get_jenkins_build_number $release $os #now works on both Oasis and FNAL CVMFS
For the former you must specify a release, an operating system flavour and a build number. The build number can be found using the latter tool (currently working only on FNAL CVMFS, not Oasis) or via the Jenkins buildmaster website:
https://buildmaster.fnal.gov/view/Nova/job/nova_SRT_slf5_release_build_output/
https://buildmaster.fnal.gov/view/Nova/job/nova_SRT_slf6_release_build_output/
Installing a UPS/UPD Product¶
Here we use the example of NovaGridUtils:
source /cvmfs/nova.opensciencegrid.org/externals/setup setup upd upd install NovaGridUtils v01.55 #extra arguments like -f NULL might be needed ups list -aK PRODUCT:VERSION:DB NovaGridUtils #Just double check that install succeeded
Publishing¶
Once the repository has been edited the new version can be published via:
nova_cvmfs_publish
Issues¶
These webpages give details related to CVMFS that have been useful for setting up the repository, and fixing issues (thanks to Dave Dykstra and Andrew Norman for all their work):
http://cernvm.cern.ch/portal/filesystem/maintain-repositories
http://cernvm.cern.ch/portal/filesystem/repository-limits
http://cernvm.cern.ch/portal/cvmfs/fix-failed-remount
https://twiki.grid.iu.edu/bin/view/Documentation/Release3/InstallCvmfs
Checking CVMFS¶
CVMFS is usual configured using auto-mount, this means that directories and files are only delived upon request. On the GPVM nodes (and typically offsite) the Oasis and FNAL CVMFS images are mounted as follows:
/cvmfs/nova.opensciencegrid.org #FNAL CVMFS
The common locations are as follows
/cvmfs/nova.opensciencegrid.org/ #base directory /cvmfs/nova.opensciencegrid.org/novasoft/slf5/novasoft/ #SLF5 /cvmfs/nova.opensciencegrid.org/novasoft/slf6/novasoft/ #SLF6 /cvmfs/nova.opensciencegrid.org/externals #external products
Example setup_nova bash function¶
function setup_nova_cvmfs_fnal { echo -en "\033[00;33mNova Offline Environment Enabled \033[00;32m(CVMFS Base)\033[00m\n" export CVMFS_DISTRO_BASE=/cvmfs/nova.opensciencegrid.org source ${CVMFS_DISTRO_BASE}/novasoft/slf5/novasoft/setup/setup_nova.sh -5 "${CVMFS_DISTRO_BASE}/novasoft/slf5/novasoft" -6 "${CVMFS_DISTRO_BASE}/novasoft/slf6/novasoft" -e "/grid/fermiapp/products/common/db:${CVMFS_DISTRO_BASE}/externals" "$@" echo "" }
[Possibly no longer relevant] Integration with Off-site Clusters¶
SMU Cluster Integration
Harvard Cluster Integration
Minnesota Cluster Integration
For more information or questions contact: Jonathan Davies (j.p.davies@sussex.ac.uk) or Andrew Norman (anorman@fnal.gov)