Project

General

Profile

Running GENIE on grid

Basic commands

  • Log into geniegpvm:
ssh geniegpvm01.fnal.gov
  • Log into geniegpvm as geniepro:
ssh geniegpvm01.fnal.gov -l geniepro (or ssh geniepro@geniegpvm01.fnal.gov)
  • Set up jobsub client:
source /cvmfs/fermilab.opensciencegrid.org/products/common/etc/setups.sh
setup jobsub_client
  • Run the job:
jobsub_submit -G [GROUP] -M --OS=SL6 --resource-provides=usage_model=[RESOURCES] [PATH_TO_SCRIPT] [SCRIPT'S ARGUMENTS]

[GROUP] = genie or fermilab
[RESOURCES] = DEDICATED,OPPORTUNISTIC or OFFSITE or DEDICATED,OPPORTUNISTIC,OFFSITE (onsite or OSG or both, respectively)
[PATH_TO_SCRIPT] = absolute or relative path to the file (file:///absolute/path/to/script or file://script)
[SCRIPT'S ARGUMENTS] = everything after path to script is assumed to be its arguments list

  • Copy files from a working node to scratch area:
source /cvmfs/fermilab.opensciencegrid.org/products/common/etc/setups.sh
setup ifdhc
ifdh cp [source] [destination]

[source] = works like regular 'cp' command, use -r for directories etc.
[destination] = "/pnfs/genie/scratch/users/..." (if run as genie group) or "/pnfs/fermilab/volatile/genie/..." (if run as fermilab group)

Example - generate splines

  • setupGENIE.sh

##### PATH TO #####

#GENIEBASE=/grid/fermiapp/genie/ # OSG can not see this area

GENIEBASE=/cvmfs/fermilab.opensciencegrid.org/genie/ # works for OSG and fermigrid

export GENIE=$GENIEBASE/rest/of/path/to/genie

##### DEPENDENCIES #####

export GUPSBASE=/cvmfs/fermilab.opensciencegrid.org/

source $GUPSBASE/products/genie/externals/setup

setup root v5_34_25a -q debug:e7:nu
setup lhapdf v5_9_1b -q debug:e7
setup log4cpp v1_1_1b -q debug:e7

##### PATHS #####

export LD_LIBRARY_PATH=$GENIE/lib:$LD_LIBRARY_PATH
export PATH=$GENIE/bin:$PATH
  • generateSplines.sh
##### SETUP GENIE #####

source /path/to/setupGENIE.sh # must be in /cvmfs/fermilab.opensciencegrid.org/genie/ for OSG

##### CREATE FOLDER FOR THE RESULTS #####

mkdir -p splines

##### RUN GENIE #####

$GENIE/bin/gmkspl -p [incident PDG] -t [target PDG] -o 'splines/mySpline.xml' [whatever other option I want to use]

##### SET UP IFDHC #####

source /cvmfs/fermilab.opensciencegrid.org/products/common/etc/setups.sh
setup ifdhc

##### COPY RESULTS TO /PNFS #####

ifdh cp -r splines /pnfs/genie/scratch/users/...
  • Submit the job:
jobsub_submit -G genie -M --OS=SL6 --resource-provides=usage_model=DEDICATED,OPPORTUNISTIC,OFFSITE file://generateSplines.sh

Example - xml splines to ROOT format

  • splinesToRoot.sh
##### SETUP GENIE #####

source /path/to/setupGENIE.sh # must be in /cvmfs/fermilab.opensciencegrid.org/genie/ for OSG

##### SET UP IFDHC #####

source /cvmfs/fermilab.opensciencegrid.org/products/common/etc/setups.sh
setup ifdhc

##### COPY XML FILES TO CONVERT #####

ifdh cp -r /pnfs/genie/scratch/users/path/to/splines .

##### CREATE FOLDER FOR THE RESULTS #####

mkdir -p rootSplines

##### CONVERT SPLINES #####

$GENIE/bin/gspl2root -p [incident PDG] -t [target PDG] -f 'splines/mySpline.xml' -o 'rootSplines/mySpline.root' [whatever other option I want to use]

##### COPY RESULTS TO /PNFS #####

ifdh cp -r rootSplines /pnfs/genie/scratch/users/...

DAG

In order to run a sequence of jobs one can use jobsub DAG:

jobsub_submit_dag -G [GROUP] [PATH_TO_DAG_FILE]

[GROUP] = genie or fermilab
[PATH_TO_DAG_FILE] = absolute or relative path to the file (file:///absolute/path/to/dag_file or file://dag_file)

  • Serial jobs within DAG file
<serial>
jobsub -n [JOBSUB_SERVER_OPTIONS] [PATH_TO_SCRIPT_1] [SCRIPT_1'S ARGUMENTS]
jobsub -n [JOBSUB_SERVER_OPTIONS] [PATH_TO_SCRIPT_2] [SCRIPT_2'S ARGUMENTS]

...
jobsub -n [JOBSUB_SERVER_OPTIONS] [PATH_TO_SCRIPT_N] [SCRIPT_N'S ARGUMENTS]
</serial>
  • Parallel jobs within DAG file
<parallel>
jobsub -n [JOBSUB_SERVER_OPTIONS] [PATH_TO_SCRIPT_1] [SCRIPT_1'S ARGUMENTS]
jobsub -n [JOBSUB_SERVER_OPTIONS] [PATH_TO_SCRIPT_2] [SCRIPT_2'S ARGUMENTS]

...
jobsub -n [JOBSUB_SERVER_OPTIONS] [PATH_TO_SCRIPT_N] [SCRIPT_N'S ARGUMENTS]
</parallel>

Please note '-n' flag which is required in jobsub dag command!!!
[JOBSUB_SERVER_OPTIONS] = like for jobsub_submit (-G [GROUP] --OS=SL6 --resource-provides=usage_model=[RESOURCES]...)
[PATH_TO_SCRIPT_i] [SCRIPT_i'S ARGUMENTS] = like for jobsub_submit

  • DAG example (onsite)
# generate cross section splines for different dynamics in parallel mode
<parallel>
jobsub -n -G genie --OS=SL6 --resource-provides=usage_model=DEDICATED,OPPORTUNISTIC file:///my/path/to/ccqe_splines.sh
jobsub -n -G genie --OS=SL6 --resource-provides=usage_model=DEDICATED,OPPORTUNISTIC file:///my/path/to/res_splines.sh
jobsub -n -G genie --OS=SL6 --resource-provides=usage_model=DEDICATED,OPPORTUNISTIC file:///my/path/to/dis_splines.sh
</parallel>
# merge splines and convert to root
<serial>
jobsub -n -G genie --OS=SL6 --resource-provides=usage_model=DEDICATED,OPPORTUNISTIC file:///my/path/to/merge.sh
jobsub -n -G genie --OS=SL6 --resource-provides=usage_model=DEDICATED,OPPORTUNISTIC file:///my/path/to/xml2root.sh
</serial>
<parallel>
# etc
</parallel>

CRON JOBS

  • Run this before you call your cron job as geniepro (I am not 100% sure, but time-life is 12h?)
voms-proxy-init -rfc -cert /opt/geniepro/geniegpvm01.fnal.gov.x509.pem -key /opt/geniepro/request.key -out /opt/geniepro/proxy -voms fermilab:/fermilab/genie/Role=Production
  • At this point, geniepro runs it at midnight (one minute before running the job)
00 00  * * * voms-proxy-init -rfc -cert /opt/geniepro/geniegpvm01.fnal.gov.x509.pem -key /opt/geniepro/request.key -out /opt/geniepro/proxy -voms fermilab:/fermilab/genie/Role=Production
  • Any cron job must be preceded by /usr/krb5/bin/kcron
00 01 * * * /usr/krb5/bin/kcron /grid/fermiapp/genie/scripts/updateBuilds.sh
  • Script must contain
export PATH=$PATH:/usr/krb5/bin/
export X509_USER_PROXY=/opt/geniepro/proxy # for geniepro

Legacy Validation

Legacy validation (src/scripts/production/batch) was translated to python and jobsub. At this moment, the code lives here: https://github.com/TomaszGolan/legacyValidation or here: /grid/fermiapp/genie/legacyValidation/.

Legacy validation 101: https://cdcvs.fnal.gov/redmine/projects/genie/wiki/Legacy_Validation