Project

General

Profile

Running Jobs on Grid using Older Release

If you are using older release (for example: S16-12-07), then you might face some problems in running jobs on grid. So in that case following hack around can help you running your jobs on grid.

Step 1

Make sure your setup_nova function is setup like this:

function setup_nova {

source /cvmfs/nova.opensciencegrid.org/novasoft/slf6/novasoft/setup/setup_nova.sh "$@" 
setup_fnal_security
cd /nova/app/users/ynitin

}

You should replace ynitin by your kerberos principal.

Step 2

Checkout the NovaGridUtils package by using command:
addpkg_svn -h NovaGridUtils

Step 3

Copy the NovaGridUtils/bin/art_sam_wrap.sh and NovaGridUtils/bin/submit_nova_art.py in your local area (the directory where Novans usually do srt_setup -a) and change art_sam_wrap.sh as:

At line 474 ( at line 473 there is written srt_setup -a, just after it) copy the following lines:

SRT_SUBDIR=`echo $SRT_SUBDIR | sed -e 's/3.1/2.6/g'`
SRT_ARCH=`echo $SRT_ARCH | sed -e 's/3.1/2.6/g'`
BFARCH=`echo $BFARCH | sed -e 's/3.1/2.6/g'`
export LD_LIBRARY_PATH=$SRT_PRIVATE_CONTEXT/lib/$SRT_SUBDIR:$SRT_PUBLIC_CONTEXT/lib/$SRT_SUBDIR:$LD_LIBRARY_PATH
export PATH=$SRT_PRIVATE_CONTEXT/bin/$SRT_SUBDIR:$SRT_PUBLIC_CONTEXT/bin/$SRT_SUBDIR:$PATH

Step 4

Change the path in the submit_nova_art.py at line 56 (which is the path to your edited art-sam_wrap.py) as this-

art_sam_wrap_cmd="art_sam_wrap.sh"

Now you have two edited files in your local area- art_sam_wrap.sh and submit_nova_art.py. Copy these two edited files inside NovaGridUtils/bin/ as well.

Then do:
source NovaGridUtils/bin/setup_local_ngu.sh

Some sanity checks before you start submitting your jobs:

Make sure that in your submit script to grid, you have no paths which has /app area (as grid doesn't see bluearc).

Also while submitting jobs the process tars up(tarball) your test release by default (but it doesn't often if the version is broken) and unrolls it in the grid.
So make sure, to remove the tarball file of your release.

Some more sanity checks:

Just to make sure

kinit -f <yourprincipal>
setup_samweb2xrootd
setup sam_web_client
setup_fnal_security

An example of grid submission script: (this is in local area, where you usually do srt_setup -a)

#!/bin/bash

WORKDIR=/nova/app/users/ynitin/art/S16-12-07/
DEST=/pnfs/nova/scratch/users/ynitin/FinalSept2017DiF/Data/test/
TAG=S16-12-07
DEFNAME=ynitin_prod_artdaq_fd_cosmic_prod3_subset10of12_isgoodrun_version19
NJOBS=1
FILESPERJOB=1
FCL=dif_reco_pid_caf.fcl
JOBNAME=nitin_dif_reco_Data
HISTTIER=dif_hist
CAFTIER=caf
DATATIER=reco

submit_nova_art.py --cvmfs --tag ${TAG} --testrel ${PWD} --maxopt --ngu_test \
--jobname ${JOBNAME} --defname ${DEFNAME} --expected_lifetime=86400 --memory=3200 \
-c ${FCL} --dest ${DEST} \
--njobs ${NJOBS} --files_per_job ${FILESPERJOB} \
--opportunistic -G nova \
--copyOut --print_jobsub --histTier ${HISTTIER} \
--cafTier=cafmakerdif:${CAFTIER} \
--outTier=out1:${DATATIER} \
--os 'SL6'

Have a happy job submitting!