Project

General

Profile

Disclaimer: Only tag releases of novasoft are currently distributed through CVMFS. It is not yet possible to setup development. Also, only OS X, SL5/6 machines are supported. Novasoft will not run on Ubuntu!

To install CVMFS on a SL5/6 compatible computer cluster please follow the instructions here .

Setting up NOvA Software on your Laptop/Desktop

The full NOvA software suite can be setup on your laptop/desktop using the Cern Virtual Machine File System (CVMFS). This system provides your laptop/desktop with an image of the NOvA base releases and all the external products that they depend on, without having to actually download the entire release and dependencies.

The way the system works is that the NOvA offline release managers build and deploy the NOvA and art software to a central server using several "flavors" corresponding to your machine (i.e. Scientific Linux or Mac OSX). This central repository is then mounted by your personal system in a read-only mode through the CVMFS client software. Only the files that you actually access are downloaded to your machine and cached locally (this means that instead of having a 150 GB download of every file in the repository, you only actually pull down 1-2 GB which allow you run your jobs or do your code development).

To use this setup you will need to install the CVMFS client software on your laptop and configure it to point to the central NOvA repository.

This document outlines the steps that are needed to install and configure your laptop for running with the CVMFS client.

The document is broken down into instructions for both Linux and Mac OSX.

Installation of the CVMFS Client

(Note: These instruction are for Mac OSX and have been verified on Mountain Lion and Mavericks, but should work on most platforms)

  • Download the client software from the cern software distribution site:

http://cernvm.cern.ch/portal/filesystem/downloads

Choose the installation that matches your operating system (i.e. Mac OSX or one of the linux variants. For Scientific Linux which is what NOvA production runs off of, use the redhat el5/6 rpms)

  • Install the client software on your machine
    • For OSX click/run the cvmfs-2.X.Y.pkg file to start the installer
    • You will be told that you do not have OSXFUSE installed (unless you actually have it installed already)
      • Follow the prompts to download the OSXFUSE installation files
      • Open/click the OSXFUSE.X.X.dmg file
      • Run the OSXFUSE installer .pkg file
      • OSXFUSE is now installed!
    • Since you now have OSXFUSE installed you need to re-run the cvmfs installer
    • Re-run the cvmfs-2.X.Y.pkg file to restart the installer
    • Follow the prompts and verify that the software installs

At this point the software needs to be configured.

  • Run the configuration script provided with the cvmfs package
sudo cvmfs_config setup

If you are not familar with the "sudo" command, it allows you to run commands as if you were the privileged user (root). When you use the sudo command you will be prompted for a password. The password you should enter is YOUR password (i.e. it is saying "are you really sure you want to do this, and are you really who you think you are") If everything is successful, the command will run and you will get no error messages (and no feed back at all as of the writing of this document, which can be confusing. It would be nice if they at least said "Everything configured properly")

Now the site (Nova) specific information needs to be added to your config files.

  • The repository we current use is "nova.opensciencegrid.org".
  • Make the following directories on your laptop:
    • /cvmfs/
    • /cvmfs/nova.opensciencegrid.org/
  • If you want to also use the cms code (or atlas code) make:
    • /cvmfs/cms.cern.ch/

(Note to make these directories you can do a "sudo mkdir -p /cvmfs/nova.opensciencegrid.org/")

  • Add the following items to your cvmfs configuration:
    • Add the site key file to /etc/cvmfs/keys/
      • Copy the file opensciencegrid.org.pub to the /etc/cvmfs/keys/ directory
      • You can find the file in the /etc/cvmfs/keys area on any machine that is already using the repository (i.e. novagpvm10.fnal.gov)
      • You can also follow the instructions from the repository maintainers to obtain their key directly
      • You can find the file in the /etc/cvmfs/config.d/ area on any machine that is already using the repository (i.e. novagpvm10.fnal.gov)
      • You can also obtain the config file from the repository maintainer's instructions (see the above url)
    • Replace your /etc/cvmfs/default.conf with the same file on novagpvm10.fnal.gov
    • Create a default.local configuration for your laptop
      • The default.local should be put in the /etc/cvmfs/ directory of your laptop
      • The file should read:
        CVMFS_REPOSITORIES=nova.opensciencegrid.org
        CVMFS_CHECK_PERMISSIONS=no
        CVMFS_HTTP_PROXY=DIRECT
        

At this point you should be ready to mount the remote repository.

  • Mount the remote repository!
    sudo mount -t cvmfs nova.opensciencegrid.org /cvmfs/nova.opensciencegrid.org
    

You should see something similar to:

rschroet@urbicandeair: ~ > sudo mount -t cvmfs nova.opensciencegrid.org /cvmfs/nova.opensciencegrid.org
CernVM-FS: running with credentials 10000:10000
CernVM-FS: loading Fuse module... done
CernVM-FS: mounted cvmfs on /cvmfs/nova.opensciencegrid.org
rschroet@urbicandeair: ~ >

And if you use the "df" command you should see:

rschroet@urbicandeair: ~ > df
Filesystem     512-blocks       Used   Available Capacity  iused   ifree %iused  Mounted on
/dev/disk0s2    488555536  455282760    32760776    94% 56974343 4095097   93%   /
devfs                 429        429           0   100%      742       0  100%   /dev
map -hosts              0          0           0   100%        0       0  100%   /net
map auto_home           0          0           0   100%        0       0  100%   /home
cvmfs2        15166603264 2091375792 13075227472    14%        0       0  100%   /cvmfs/nova.opensciencegrid.org

At this point you can now take a look at the NOvA base releases. Simply:

 ~ > ls /cvmfs/nova.opensciencegrid.org/

Another useful command is the following. It updates and remounts the cvmfs, to pick up any recently published changes.

sudo cvmfs_talk -i nova.opensciencegrid.org <http://nova.opensciencegrid.org/> remount

Using the Base Release

The following procedure only works on SL5/6 compatible machines. Mac OS X and Ubuntu are currently not supported.

Using the base release is as simple as setting up the normal nova software on the interactive login nodes at fermilab.

You will still need to have a couple of setup lines in your .bash_profile or .bashrc file that look like:

function setup_novaoffline_cvmfs {
  echo -en "Enabling Nova Offline Environment (CVMFS Base Distro)\n" 
################################################
# This is the prefix to your site's cvmfs area.
# Most sites will use /cvmfs as the base of the path followed by 
# the specific repository that your code is stored in
# (here it is nova.opensciencegrid.org) followed by the path to the actual software 
# (here the nova code is under /nova/)
  export CVMFS_DISTRO_BASE=/cvmfs/nova.opensciencegrid.org
  export CVMFS_DISTRO_SLF=$CVMFS_DISTRO_BASE/novasoft/slf6/novasoft
################################################
# Next source the SRT setup files to make the art software available 
  source $CVMFS_DISTRO_SLF/srt/srt.sh
################################################
# Then set the path for the external products
  export EXTERNALS=$CVMFS_DISTRO_BASE/externals
  export PRODUCTS=$CVMFS_DISTRO_BASE/externals
################################################
# Now source the actual NOvA srt setup to make everything available
  source $CVMFS_DISTRO_SLF/setup/setup_nova.sh -e $PRODUCTS -6 $CVMFS_DISTRO_SLF "$@" 
################################################
# Finally change to your personal test release and set it as the
# current test release (this step is optional)
  pushd ~/NOvATestRel
  srt_setup -a
################################################
# We are done!
  echo -en "SRT Test Release set to: $PWD" 
}

And you will need to make a test release on your machine where you can checkout and edit code. (mine is ~/NOVA-OFFLINE). At this point you can start developing and running code.