Project

General

Profile

Procedure for GR when closing the dataset

Tagging the GR database

What to check before tagging

Mistakes happen - sometimes cronjobs fail, sometimes code has been commented out - and we want to make sure the appropriate runs are in the database before we make a new tag. To do this, while you are logged into a gpvm, log into novadqgpvm02 as novadq:

ssh novadq@novadqgpvm02.fnal.gov

Once you are logged in, do

crontab -l

and look for the list of jobs that say "#Good Runs - Database". Go to the directory where the LoadDQValidity scripts are being run. For now they are located at

/home/novadq/Releases/Nearline/S18-09-02/Commissioning/GoodRuns/{Near,Far}Det/database

This will change if the release changes. The LoadDQValidity scripts run daily and create a csv that is a copy of the database. For the ND, it is databasecopyND.csv. For the FD, it is testdqcopy.csv. Open these files, scroll to the bottom, and make sure the last run in the database is consistent with a run that has completed within the last day. If the last run in that file is older than the last day, the database is not current. If the database is current, skip the next section - you can tag the database.

If the database is not current

To find out if the cron jobs have failed, check

/home/novadq/Releases/Nearline/logs/GoodRuns/database/LoadDQValidityND.log
/home/novadq/Releases/Nearline/logs/GoodRuns/database/LoadDQValidity.log

The first log is for the ND, the second for the FD. Another tip to check whether the cron jobs have run and updated the appropriate files is to

ls -ltr

in the database directories. For the ND, DQValidityTableND.csv, databasecopyND.csv, and dqvaliditynewND.csv, and for the FD, DQValidityTable.csv, testdqcopy.csv, and dqvaliditynew.csv should all be updated daily. If these files are not current, you will need to run the entire LoadValidityDQ script for the appropriate detector:

bash /home/novadq/Releases/Nearline/S18-09-02/Commissioning/GoodRuns/NearDet/database/LoadDQValidityND.sh
bash /home/novadq/Releases/Nearline/S18-09-02/Commissioning/GoodRuns/FarDet/database/LoadDQValidity.sh

If these files are current, but the database is not current, you will have to write to the database manually.

source /home/novadq/.bashrc
setup_nova -r $REL -b maxopt
cd /home/novadq/Releases/Nearline/$REL/
srt_setup -a
cd Commissioning/GoodRuns/FarDet/database/

export NOVADBHOST=ifdbprod.fnal.gov
export NOVADBNAME=nova_prod
export NOVADBPORT=5433
export NOVADBUSER=nova_reader
export NOVADBPWDFILE=~/.pgpass
export NOVADBWSPWDFILE=~/condbprod.pwd
export NOVADBWSURL=http://novacon-data.fnal.gov:8091/NOvACon/v2_2b/app/

Then, for ND

writeValidityCSVtoDB NearDet data tables/neardet_dq.xml dqvaliditynewND.csv

And for FD

writeValidityCSVtoDB FarDet data tables/DQValidity.xml dqvaliditynew.csv

How to tag the database

Run the following commands, substituting the appropriate tag number. The 'v' must be included in the tag.

tagValidityTableInDB -h ifdbprod.fnal.gov -n nova_prod -p 5433 -d NearDet -f tables/neardet_dq.xml -t v4.2
tagValidityTableInDB -h ifdbprod.fnal.gov -n nova_prod -p 5433 -d FarDet -f tables/DQValidity.xml -t v7.1

Pushing the GR database to SAM

Accessing the SAM database

Only a few people are able to push tables to the database. Go to Service Now, click "request something," and ask for nova_sam_gr permissions for the sam_nova_prd and samtest databases.

Easy way to test your access is, in python, interactively:

import psycopg2, psycopg2.extras
psycopg2.connect("dbname=samtest host=cspgsdev.fnal.gov port=5432")
psycopg2.connect("dbname=sam_nova_prd host=sampgsdb01.fnal.gov port=5433")

As of 11/7/2019 the host and ports above are the correct ones for accessing each database. (Tested again on 3/25/2020, still correct.)

If your connect commands are successful, you'll get output like:

>>> psycopg2.connect("dbname=sam_nova_prd host=sampgsdb01.fnal.gov port=5433")
<connection object at 0x7fec9b3a8e88; dsn: 'dbname=sam_nova_prd host=sampgsdb01.fnal.gov port=5433', closed: 0>

If they are unsuccessful, you'll get output like:

psycopg2.OperationalError: could not connect to server: Connection refused
    Is the server running on host "cspgsprd1.fnal.gov" (131.225.107.59) and accepting
    TCP/IP connections on port 5432?

If the connection is failing, you will need to ask the database admins (via a ticket) if the database has been moved to a new host etc. If this is the case you will also need to update the script, linked below.

Using the script

First, run a test to make sure that you're going to push the right thing to SAM. Do:

python getGoodRuns4SAM.py -D fardet -T v6.1 --test

where -D is either fardet or neardet, -T is the tag of the Good Runs database that you want to push, and --test indicates that you will not be pushing to the SAM database. The most recent tags should be here: List of Validated FD subruns

When you test, the script will write out a csv ("goodRuns4SAM.csv") which is what will be pushed to SAM when you rerun. Open that file, and again scroll to the bottom and check that the last run is recent. If this is not the case, then the database was not updated before creating the tag, and you will need to update the database and re-tag it.

If the last run is recent, rerun getGoodRuns4SAM.py without the --test flag.

Deleting definitions (e.g. to update one)

swdeld isgood_nd_prod5
swdeld isgood_fd_prod5
swdeld isgood_prod5

Making the definitions

samweb create-definition isgood_nd_prod5 "novagr_good true with novagr_tag v4.0" 
samweb create-definition isgood_fd_prod5 "novagr_good true and novagr_ngood_cont_diblock > 3 with novagr_tag v7.0" 
samweb create-definition isgood_prod5 "defname: isgood_nd_prod5 or defname: isgood_fd_prod5" 

Replace with the current production campaign number and ND and FD GR database tags as needed.

Resources

Tech note
Script to push to SAM DB
Example of definition updating