Project

General

Profile

History

Novaart ยป

Preamble

This site is mainly intended as a reference manual for super users (i.e. current and past NOvA offline release managers) to document any details necessary for maintaining, pubishing and distributing software via CVMFS.

NOvA Software in CVMFS

The full nova-art/nusoft distribution has been made fully relocatable so that it can be hosted off of a number of different platforms that allow for code distribution. This allows the nova software to be made available offsite to remote computing clusters and to individual uses who want to easily run NOvA jobs from their laptop or desktop computers (without the hassle of maintaining a base release).

Currently (Feb 2016) we have distributions of the NOvA software installed on a Cern Virtual Machine File System (CVMFS) servers which distributes the code to various sites. CVMFS is the same technology that Atlas and CMS use to distribute their software distributions because it features a just-in-time file delivery scheme with local caching (meaning copy only those files that you actually use to your local machine when you use them). This makes the system especially attractive when using the NOvA software system, since the full base release + external packages is over 150 GB, while the actual run time foot print of most jobs is only 1.5-2 GB.

Our CVMFS is hosted and maintained at Fermilab. We formerly used the Oasis CVMFS server (not hosted or maintained at Fermilab), but stopped using it in the summer of 2015. We made the switch to improve turnaround times for handling problems, and also speed up the process of publishing our software to CVMFS.

The following sections detail what has been done to setup the NOvA software on the CVMFS servers and what needs to be done to use it at different remote sites.

Overview

Overall, the process for publish to CVFMS is as follows:

  1. Login to the CVMFS server
  2. Put the repository into a transaction
  3. copy the software to be published to the destination (for example through upd)
  4. Optionally verify that the copy was successful (but highly recommended)
  5. publish updated repository

Login

Access to super users is controlled via a 

~/.k5login
file. Just ask for your principle to be added by someone that already has access.

Use this command to log in:

ssh -l cvmfsnova oasiscfs.fnal.gov

Workflow (modifying / adding / publishing)

Once you have logged on to the FNAL CVMFS server you MUST put the repository in to transaction before making any modifications / additions / deletes to the repository. If you don't you can put the server into a non-trivial state, and may need to remount the repository (there are some links later to help deal with such situations that can occur).

Transaction start / check

Use these commands to check if a transaction has been started / start a transaction:

nova_cvmfs_check

nova_cvmfs_transaction

NB. These are just wrappers for this command:

cvmfs_server transaction nova.opensciencegrid.org

Software Locations

Common locations:

/cvmfs/nova.opensciencegrid.org/                        #base directory                                              
/cvmfs/nova.opensciencegrid.org/novasoft/slf5/novasoft/ #SLF5                                                        
/cvmfs/nova.opensciencegrid.org/novasoft/slf6/novasoft/ #SLF6                                                        
/cvmfs/nova.opensciencegrid.org/externals               #external products

Copying release from Jenkins

nova_cvmfs_copy_release_from_jenkins $release $os $buildnum 

nova_cvmfs_get_jenkins_build_number $release $os            #now works on both Oasis and FNAL CVMFS

For the former you must specify a release, an operating system flavour and a build number. The build number can be found using the latter tool (currently working only on FNAL CVMFS, not Oasis) or via the Jenkins buildmaster website:

https://buildmaster.fnal.gov/view/Nova/job/nova_SRT_slf5_release_build_output/
https://buildmaster.fnal.gov/view/Nova/job/nova_SRT_slf6_release_build_output/

Installing a UPS/UPD Product

Here we use the example of NovaGridUtils:

source /cvmfs/nova.opensciencegrid.org/externals/setup
setup upd
upd install NovaGridUtils v01.55 #extra arguments like -f NULL might be needed
ups list -aK PRODUCT:VERSION:DB NovaGridUtils #Just double check that install succeeded

Publishing

Once the repository has been edited the new version can be published via:

nova_cvmfs_publish

Issues

These webpages give details related to CVMFS that have been useful for setting up the repository, and fixing issues (thanks to Dave Dykstra and Andrew Norman for all their work):

http://cernvm.cern.ch/portal/filesystem/maintain-repositories
http://cernvm.cern.ch/portal/filesystem/repository-limits
http://cernvm.cern.ch/portal/cvmfs/fix-failed-remount
https://twiki.grid.iu.edu/bin/view/Documentation/Release3/InstallCvmfs

Checking CVMFS

CVMFS is usual configured using auto-mount, this means that directories and files are only delived upon request. On the GPVM nodes (and typically offsite) the Oasis and FNAL CVMFS images are mounted as follows:

/cvmfs/nova.opensciencegrid.org       #FNAL CVMFS

The common locations are as follows

/cvmfs/nova.opensciencegrid.org/                        #base directory                                              
/cvmfs/nova.opensciencegrid.org/novasoft/slf5/novasoft/ #SLF5                                                        
/cvmfs/nova.opensciencegrid.org/novasoft/slf6/novasoft/ #SLF6                                                        
/cvmfs/nova.opensciencegrid.org/externals               #external products

Example setup_nova bash function

function setup_nova_cvmfs_fnal {

    echo -en "\033[00;33mNova Offline Environment Enabled \033[00;32m(CVMFS Base)\033[00m\n" 
    export CVMFS_DISTRO_BASE=/cvmfs/nova.opensciencegrid.org
    source ${CVMFS_DISTRO_BASE}/novasoft/slf5/novasoft/setup/setup_nova.sh -5 "${CVMFS_DISTRO_BASE}/novasoft/slf5/novasoft" -6 "${CVMFS_DISTRO_BASE}/novasoft/slf6/novasoft" -e "/grid/fermiapp/products/common/db:${CVMFS_DISTRO_BASE}/externals" "$@" 
    echo "" 

}

[Possibly no longer relevant] Integration with Off-site Clusters

SMU Cluster Integration
Harvard Cluster Integration
Minnesota Cluster Integration

For more information or questions contact: Jonathan Davies () or Andrew Norman ()