Joining the Production Group

There are a few steps in joining the production group, mostly not done by you.

Get production submission privileges (updated May 7, 2019)

  1. Navigate to and click on "Request Something"
  2. In "Request Catalog Categories" on the right-hand side of the page click on "Accounts and Access"
  3. In "Matching items" click on "Computer accounts/computing permissions for an experiment"
  4. Choose "E-929 (NOvA)" for Affiliation/Experiment, "No" for Computing/IT Professional Access Only, and "Production" for Roles

If your request gets stuck in limbo, or the person who triages your ticket doesn't know where it should go, mention the "VOMS novapro accounting group", which should take you to the right people.

Get a CILogon certificate (if you don't already have one)

Follow the instructions here (they're from the DUNE experiment, but there's nothing DUNE-specific in them):
This will enable you to look at job logfiles from the FIFEmon pages in your browser.


Join the nova_production mailing list by sending an email to with the following body:

subscribe nova_production

Also join the #production group on Slack.

Ask a production convener to...

Useful Aliases

The following are some aliases you may find useful in the course of submitting production jobs.

alias swcf="samweb count-files" 
alias swts="samweb take-snapshot" 
alias swcdf="samweb count-definition-files" 
alias swlf="samweb list-files" 
alias swlp="samweb list-projects" 
alias swldf="samweb list-definition-files" 
alias swdd="samweb describe-definition" 
alias swdeld="samweb delete-definition" 
alias swgm="samweb get-metadata" 
alias swldesc="samweb file-lineage descendants" 
alias swlanc="samweb file-lineage ancestors" 

alias pjs='prodjob-summary | grep -v keepup'
alias pjst='prodjob-summary | grep testjob'
alias pjsk='prodjob-summary | grep keepup'
alias pjsall='prodjob-summary'

alias sna=''
alias snat=' --test_submission --test_queue'
alias ups_list="ups list -aK+" 
alias ups_list_novapro="ups list -aK+ novaproduction" 
alias ups_list_ngu="ups list -aK+ NovaGridUtils" 

Submit your first production job

1. Get Fermilab accounts ready

Have both Fermilab Service Account and Kerberos Computing Account ready. The initial passwords need to be obtained by calling Fermilab Service Desk (630-840-2345). To change both of them, you can follow:

How to change your Kerberos password instructions:
How to change your Services account password instructions:

2. Install and set up Kerberos software

Kerberos Configuration File can be found at:

For Macintosh users, more details can be found at:

Log in to ECL

3. Submit a test job

Authenticate to Kerberos on your local PC. (Generally, this needs to be done everyday, as the authentication is only stored on your local machine for up to 24 hours.)

kinit -f

Ask production conveners to have a configure file for submission. Mine is:


Ssh to using your Fermilab/Kerberos Account username.

ssh -o "GSSAPIAuthentication=yes” <your_username>

Source the script setting up the environment for nova production. You can add it to your bash profile (~/.bash_profile) as an alias for your convenience.

source /cvmfs/ "$@" 

Set up Nova environment (equivalent to the above command):


Create a folder:

mkdir -p prod/testbeam

mkdir -p prod/testbeam

Copy the configure file to your folder and navigate to this folder on the remote node.

cp /nova/app/users/lasquith/PROD5.1/TESTBEAM/artdaq_to_pclist_testbeam.cfg ./prod/testbeam

For your first time login, set up the security: get the production submission privileges following and run:

setup_fnal_security -f

Run this python script to submit the test job: -f artdaq_to_pclist_testbeam.cfg --test_submission --test_queue |tee testsub_artdaq_to_pclist_testbeam.log

You can check submitted jobs’ status by:


4. Submit real production jobs

Copy a new cfg file from the one for testing. Uncomment the line of “continue_project”.

Run this python script again without “—test_submission”, to submit the real job: -f artdaq_to_pclist_testbeam.cfg --test_queue |tee testsub_artdaq_to_pclist_testbeam.log

You can check submitted jobs’ status by:


5. Add New Entry on ECL

After submitting official jobs on servers, add corresponding ECL entries.

  1. Click New Entry in the left most column
  2. In the form, choose `Production Form Checklist`
  3. Add the following Tags : Calib , MC , TB
  4. Entry Subject : TB pclist
  5. Check mark Add category and tags
  6. Check mark input defn prestaged
  7. Write the path of the of your test_jobs/xxxx_xxxx folder where the output test files exists
  8. Copy-paste the .cfg file from the convener
  9. Copy-paste the submission log that should be saved in the artdaq_to_pclist_testbeam.log
  10. The ends