Procedure for GR when closing the dataset » History » Version 6
- Table of contents
- Tagging the GR database
- Pushing the GR database to SAM
Tagging the GR database¶
What to check before tagging¶
Mistakes happen - sometimes cronjobs fail, sometimes code has been commented out - and we want to make sure the appropriate runs are in the database before we make a new tag. To do this, while you are logged into a gpvm, log into novadqgpvm02 as novadq:
Once you are logged in, do
and look for the list of jobs that say "#Good Runs - Database". Go to the directory where the LoadDQValidity scripts are being run. For now they are located at
This will change if the release changes. The LoadDQValidity scripts run daily and create a csv that is a copy of the database. For the ND, it is databasecopyND.csv. For the FD, it is testdqcopy.csv. Open these files, scroll to the bottom, and make sure the last run in the database is consistent with a run that has completed within the last day. If the last run in that file is older than the last day, the database is not current. If the database is current, skip the next section - you can tag the database.
If the database is not current¶
To find out if the cron jobs have failed, check
The first log is for the ND, the second for the FD. Another tip to check whether the cron jobs have run and updated the appropriate files is to
in the database directories. For the ND, DQValidityTableND.csv, databasecopyND.csv, and dqvaliditynewND.csv, and for the FD, DQValidityTable.csv, testdqcopy.csv, and dqvaliditynew.csv should all be updated daily. If these files are not current, you will need to run the entire LoadValidityDQ script for the appropriate detector:
bash /home/novadq/Releases/Nearline/S18-09-02/Commissioning/GoodRuns/NearDet/database/LoadDQValidityND.sh bash /home/novadq/Releases/Nearline/S18-09-02/Commissioning/GoodRuns/FarDet/database/LoadDQValidity.sh
If these files are current, but the database is not current, you will have to write to the database manually.
source /home/novadq/.bashrc setup_nova -r $REL -b maxopt cd /home/novadq/Releases/Nearline/$REL/ srt_setup -a cd Commissioning/GoodRuns/FarDet/database/ export NOVADBHOST=ifdbprod.fnal.gov export NOVADBNAME=nova_prod export NOVADBPORT=5433 export NOVADBUSER=nova_reader export NOVADBPWDFILE=~/.pgpass export NOVADBWSPWDFILE=~/condbprod.pwd export NOVADBWSURL=http://novacon-data.fnal.gov:8091/NOvACon/v2_2b/app/
Then, for ND
writeValidityCSVtoDB NearDet data tables/neardet_dq.xml dqvaliditynewND.csv
And for FD
writeValidityCSVtoDB FarDet data tables/DQValidity.xml dqvaliditynew.csv
How to tag the database¶
Run the following commands, substituting the appropriate tag number. The 'v' must be included in the tag.
tagValidityTableInDB -h ifdbprod.fnal.gov -n nova_prod -p 5433 -d NearDet -f tables/neardet_dq.xml -t v4.2 tagValidityTableInDB -h ifdbprod.fnal.gov -n nova_prod -p 5433 -d FarDet -f tables/DQValidity.xml -t v7.1
Pushing the GR database to SAM¶
Accessing the SAM database¶
Only a few people are able to push tables to the database. Go to Service Now, click "request something," and ask for nova_sam_gr permissions for the sam_nova_prd and samtest databases.
Easy way to test your access is, in python, interactively:
import psycopg2, psycopg2.extras psycopg2.connect("dbname=samtest host=cspgsdev.fnal.gov port=5432") psycopg2.connect("dbname=sam_nova_prd host=sampgsdb01.fnal.gov port=5433")
As of 11/7/2019 the host and ports above are the correct ones for accessing each database. (Tested again on 3/25/2020, still correct.)
If your connect commands are successful, you'll get output like:
>>> psycopg2.connect("dbname=sam_nova_prd host=sampgsdb01.fnal.gov port=5433") <connection object at 0x7fec9b3a8e88; dsn: 'dbname=sam_nova_prd host=sampgsdb01.fnal.gov port=5433', closed: 0>
If they are unsuccessful, you'll get output like:
psycopg2.OperationalError: could not connect to server: Connection refused Is the server running on host "cspgsprd1.fnal.gov" (22.214.171.124) and accepting TCP/IP connections on port 5432?
If the connection is failing, you will need to ask the database admins (via a ticket) if the database has been moved to a new host etc. If this is the case you will also need to update the script, linked below.
Using the script¶
First, run a test to make sure that you're going to push the right thing to SAM. Do:
python getGoodRuns4SAM.py -D fardet -T v6.1 --test
where -D is either fardet or neardet, -T is the tag of the Good Runs database that you want to push, and --test indicates that you will not be pushing to the SAM database. The most recent tags should be here: List of Validated FD subruns
When you test, the script will write out a csv ("goodRuns4SAM.csv") which is what will be pushed to SAM when you rerun. Open that file, and again scroll to the bottom and check that the last run is recent. If this is not the case, then the database was not updated before creating the tag, and you will need to update the database and re-tag it.
If the last run is recent, rerun getGoodRuns4SAM.py without the --test flag.
Deleting definitions (e.g. to update one)¶
swdeld isgood_nd_prod5 swdeld isgood_fd_prod5 swdeld isgood_prod5
Making the definitions¶
samweb create-definition isgood_nd_prod5 "novagr_good true with novagr_tag v4.0" samweb create-definition isgood_fd_prod5 "novagr_good true and novagr_ngood_cont_diblock > 3 with novagr_tag v7.0" samweb create-definition isgood_prod5 "defname: isgood_nd_prod5 or defname: isgood_fd_prod5"
Replace with the current production campaign number and ND and FD GR database tags as needed.