Project

General

Profile

Joining the Production Group

There are a few steps in joining the production group, mostly not done by you.

Get production submission privileges (updated May 7, 2019)

  1. Navigate to https://fermi.servicenowservices.com/ and click on "Request Something"
  2. In "Request Catalog Categories" on the right-hand side of the page click on "Accounts and Access"
  3. In "Matching items" click on "Computer accounts/computing permissions for an experiment"
  4. Choose "E-929 (NOvA)" for Affiliation/Experiment, "No" for Computing/IT Professional Access Only, and "Production" for Roles

If your request gets stuck in limbo, or the person who triages your ticket doesn't know where it should go, mention the "VOMS novapro accounting group", which should take you to the right people.

Get a CILogon certificate (if you don't already have one)

Follow the instructions here (they're from the DUNE experiment, but there's nothing DUNE-specific in them): https://web.fnal.gov/project/LBNF/Documentation/SitePages/Get%20a%20CI%20Logon%20Certificate.aspx
This will enable you to look at job logfiles from the FIFEmon pages in your browser.

Communication

Join the nova_production mailing list by sending an email to listserv.fnal.gov with the following body:

subscribe nova_production

Also join the #production group on Slack.

Ask a production convener to...

Useful Aliases

The following are some aliases you may find useful in the course of submitting production jobs.

alias swcf="samweb count-files" 
alias swts="samweb take-snapshot" 
alias swcdf="samweb count-definition-files" 
alias swlf="samweb list-files" 
alias swlp="samweb list-projects" 
alias swldf="samweb list-definition-files" 
alias swdd="samweb describe-definition" 
alias swdeld="samweb delete-definition" 
alias swgm="samweb get-metadata" 
alias swldesc="samweb file-lineage descendants" 
alias swlanc="samweb file-lineage ancestors" 

alias pjs='prodjob-summary | grep -v keepup'
alias pjst='prodjob-summary | grep testjob'
alias pjsk='prodjob-summary | grep keepup'
alias pjsall='prodjob-summary'

alias sna='submit_nova_art.py'
alias snat='submit_nova_art.py --test_submission --test_queue'
alias ups_list="ups list -aK+" 
alias ups_list_novapro="ups list -aK+ novaproduction" 
alias ups_list_ngu="ups list -aK+ NovaGridUtils" 

Submit your first production job

1. Get Fermilab accounts ready

Have both Fermilab Service Account and Kerberos Computing Account ready. The initial passwords need to be obtained by calling Fermilab Service Desk (630-840-2345). To change both of them, you can follow:

How to change your Kerberos password instructions:
https://fermi.service-now.com/kb_view.do?sysparm_article=KB0010628
How to change your Services account password instructions:
https://fermi.service-now.com/kb_view.do?sysparm_article=KB0010542

2. Install and set up Kerberos software

https://cdcvs.fnal.gov/redmine/projects/nova-cvs/wiki/Accessing_NOvA_software_at_FNAL

Kerberos Configuration File can be found at:
https://authentication.fnal.gov/krb5conf/

For Macintosh users, more details can be found at:
https://fermi.servicenowservices.com/kb_view.do?sysparm_article=KB0011294

Log in to ECL
https://dbweb6.fnal.gov:8443/ECL/novapro/E/index

3. Submit a test job

Authenticate to Kerberos on your local PC. (Generally, this needs to be done everyday, as the authentication is only stored on your local machine for up to 24 hours.)

kinit -f junzeliu@FNAL.gov

Ask production conveners to have a configure file for submission. Mine is:

/nova/app/users/lasquith/PROD5.1/TESTBEAM/artdaq_to_pclist_testbeam.cfg

Ssh to novagpvm02.fnal.gov using your Fermilab/Kerberos Account username.

ssh -o "GSSAPIAuthentication=yes” <your_username>@novagpvm02.fnal.gov

Source the script setting up the environment for nova production. You can add it to your bash profile (~/.bash_profile) as an alias for your convenience.

source /cvmfs/nova.opensciencegrid.org/novasoft/slf6/novasoft/setup/setup_nova.sh "$@" 

Set up Nova environment (equivalent to the above command):

setup_nova

Create a folder:

mkdir -p prod/testbeam

mkdir -p prod/testbeam

Copy the configure file to your folder and navigate to this folder on the remote node.

cp /nova/app/users/lasquith/PROD5.1/TESTBEAM/artdaq_to_pclist_testbeam.cfg ./prod/testbeam

For your first time login, set up the security: get the production submission privileges following https://cdcvs.fnal.gov/redmine/projects/nova-production/wiki/Joining_the_Production_Group and run:

setup_fnal_security -f

Run this python script to submit the test job:

submit_nova_art.py -f artdaq_to_pclist_testbeam.cfg --test_submission --test_queue |tee testsub_artdaq_to_pclist_testbeam.log

You can check submitted jobs’ status by:

prodjob-summary

4. Submit real production jobs

Copy a new cfg file from the one for testing. Uncomment the line of “continue_project”.

Run this python script again without “—test_submission”, to submit the real job:

submit_nova_art.py -f artdaq_to_pclist_testbeam.cfg --test_queue |tee testsub_artdaq_to_pclist_testbeam.log

You can check submitted jobs’ status by:

prodjob-summary

5. Add New Entry on ECL

After submitting official jobs on servers, add corresponding ECL entries.

  1. Click New Entry in the left most column
  2. In the form, choose `Production Form Checklist`
  3. Add the following Tags : Calib , MC , TB
  4. Entry Subject : TB pclist
  5. Check mark Add category and tags
  6. Check mark input defn prestaged
  7. Write the path of the of your test_jobs/xxxx_xxxx folder where the output test files exists
  8. Copy-paste the .cfg file from the convener
  9. Copy-paste the submission log that should be saved in the artdaq_to_pclist_testbeam.log
  10. The ends