NOvA¶
AWS tests¶
Here you can find a summary of the AWS tests ran by the OPOS team. Test started being ran by OPOS on January 2016.
Offsite Probe tests¶
Here you can find details about this activity.
NOvA Period and Epoch Naming¶
https://cdcvs.fnal.gov/redmine/projects/novaart/wiki/Period_and_Epoch_Naming
Production Systems Health Check¶
Follow the instructions available here to monitor the NOvA production systems.
Relevant Monitoring Links¶
To monitor the NOvA Offline Systems, you can use the following links:
NOvA SAM station
http://samweb.fnal.gov:8480/station_monitor/nova/stations/nova/projects
- http://novasamgpvm01.fnal.gov:8888/fts/status
- http://novasamgpvm02.fnal.gov:8888/fts/status
- http://novasamgpvm02.fnal.gov:8889/fts/status
- http://novasamgpvm03.fnal.gov:8888/fts/status
- http://novasamgpvm04.fnal.gov:8888/fts/status
NOvA Fifemon
http://fifemon.fnal.gov/monitor/experiment/nova
NOvA Fifemon Batch
http://fifemon.fnal.gov/monitor/experiment/nova/batch
NOvA DAQ2RawDigits Base Releases
http://nusoft.fnal.gov/nova/production/sandbox/d3_raw2root_ana.html
You can also check overall monitoring tools in: https://cdcvs.fnal.gov/redmine/projects/offline_production_operations_service/wiki/Hello_Team#Hello-Team
Data Tiers¶
Naming datasets¶
For all the processing stages of NOvA Offline Production (Keepup), there are input and output dataset defined by SAM dimensions. In order to follow a standard, we suggest to use this pattern for input or output datasets:
prod_[data_tier]_[nova_release]_[detector_id]_[stream]_keepup.[additional_data]
For example:
- Raw2Root input dataset: prod_raw_FA14-10-03_nd_others_keepup.2015-04-26
- Reconstruction input dataset: prod_artdaq_S15-02-05_nd_numi_keepup.2015-04-18
- Reconstruction output dataset: prod_reco_S15-02-05_nd_numi_keepup.r10496_r10798
Raw2Root KeepUp¶
Raw2Root is the first stage for NOvA Offline Production. It will take raw data to produce ROOT files. The output files for this stage will belong to artdaq tier.
SAM Dataset dimensions¶
Dimensions for datasets (draining dataset) that will be used as input of this stage:
file_type = 'importedDetector' and
data_tier = 'raw' and
not isparentof: ( data_tier = 'artdaq' and
DAQ2RawDigit.base_release '${RELEASE}' ) and
Online.Detector = '${DET}' and
Online.SubRunEndTime >= '${START_DAY_SECONDS}' and
Online.SubRunEndTime <= '${END_DAY_SECONDS}' and
$STREAMDIM and
Online.TotalEvents > '0' and
file_size < '1288490189' and
Online.SubRunEndTime > '1432098000'
minus NOVA.ProductionSkip true
The value of
$STREAMDIM
will depend on the specified stream. The stream options are Numi, Cosmics and Others.
Stream | $STREAMDIM |
---|---|
Numi | Online.Stream = 0 |
Cosmics | Online.Stream = 2 |
Others | Online.Stream != 0 and Online.Stream != 2 and Online.Stream != 4 |
Raw2Root KeepUp submission¶
Raw2Root has two steps: input dataset generation and job submission. Raw files are collected every day in the NOvA Detectors, Raw2Root needs to look for the files that were created in the last previous days and process them. The script Raw2RootKeepUp.sh
will look for the files created up to 1 month ago iterating over odd dates. A maximum of 16 datasets will be created per Raw2Root running.
Raw2RootKeepUp.sh
Usage:
$ Raw2RootKeepUp.sh <arguments>
Arguments¶
You must specify the name of the detector which the files belong to, the name of the stream and the NOvA release you want to use to process those files.
--det DETECTORNAME Detector name: fardet or neardet --stream STREAM SAM dataset definition to run over --release NOVARELEASE NOvA Release under Raw2Root will process files
Raw2Root KeepUp as cron jobs¶
Raw2Root Keepup involves a set of cron jobs running daily. A single cron job will process one detector and one stream: NuMI, Cosmics or Others. It will start with Far Detector files, processing NuMI, Cosmics and Others streams separately with S14-08-19
release. It will continue with Near Detector files, following the same order for streams with FA14-10-03
release.There are 6 cron jobs placed in a crontab to execute Raw2Root KeepUp.
Raw2Root processing will start at 01:00 a.m.
Crontab list output (novapro@novagpvm01.fnal.gov)
00 01 * * * /nova/app/home/novapro/KeepUp_opos/Raw2Root_KeepUp/Raw2RootKeepUp.sh --det fardet --stream numi --release S15-03-11
00 03 * * * /nova/app/home/novapro/KeepUp_opos/Raw2Root_KeepUp/Raw2RootKeepUp.sh --det fardet --stream cosmics --release S15-03-11
00 05 * * * /nova/app/home/novapro/KeepUp_opos/Raw2Root_KeepUp/Raw2RootKeepUp.sh --det fardet --stream others --release S15-03-11
00 07 * * * /nova/app/home/novapro/KeepUp_opos/Raw2Root_KeepUp/Raw2RootKeepUp.sh --det neardet --stream numi --release FA14-10-03
00 09 * * * /nova/app/home/novapro/KeepUp_opos/Raw2Root_KeepUp/Raw2RootKeepUp.sh --det neardet --stream cosmics --release FA14-10-03
00 11 * * * /nova/app/home/novapro/KeepUp_opos/Raw2Root_KeepUp/Raw2RootKeepUp.sh --det neardet --stream others --release FA14-10-03
Notes¶
- Since November 2014, Near Detector Raw2Root KeepUp Processing is done under FA14-10-03. Due to an error in geometry detector, files from run number = 10377 up to 10620 were reprocessed. A set of raw files have two children files under different releases.
- Since December 12th, 2014, new dimensions for Raw2Root input datasets were defined. After an error reported for NOvA production, dimensions were fixed.
- Since April, 2015, new dimensions for Raw2Root input datasets were defined. After a request from NOvA production,
SubRunStartTime
metadata field was replaced bySubRunEndTime
. - Since end of May, 2015, all files with size greater than 1.2 GB are not being R2R as requested by the collaboration.
- Since end of May, 2015, Far Detector Raw2Root KeepUp Processing is done under S15-03-11. A set of raw files have two children files under different releases.
Raw2Root Backprocessing¶
When a Raw2Root Backprocessing request comes from NOvA experiment, the input dataset is created using the NOvA Editor.
Reconstruction Keepup¶
Reconstruction stage takes files from the artdaq tier and will produce output files in reco, caf tiers.
SAM Dataset dimensions¶
Dimensions for datasets (draining dataset) that will be used as input of this stage:
file_type = 'importedDetector' and
data_tier = 'artdaq' and
Online.Detector = '${DET}' and
Online.Partition = 1 and
Online.SubRunStartTime >= '${START_DAY_SECONDS}' and
Online.SubRunStartTime <= '${END_DAY_SECONDS}' and
Online.Stream = $STREAM and not
isparentof: ( data_tier = 'reco')
The list of values for $STREAM can be checked here: http://cdcvs.fnal.gov/lxr/nova/source/Online/pkgs/DAQDataFormats/cxx/include/TriggerDefines.h#241
Scripts¶
Reconstruction scripts are located in: /nova/app/home/novapro/KeepUp_oog/Reconstruction_KeepUp
cron_KeepUp.sh
This script has those input parameters: Detector name: {fardet | neardet}, Stream: {0 | 2 | -1 }, Release
This script will export variables and execute the dataset generation script for Reconstruction KeepUp.
ReconstructionKeepUp.sh
This script has those input parameters: Detector name: {fardet | neardet}, Stream: {0 | 2 | -1 }, Release
This script will setup the environment and build the datasets using the dimensions defined before.
Once the datasets are ready, this script will execute the submission script submitJob_keepup.sh.
submitJob_fd_keepup.sh | submitJob_nd_keepup.sh
This script has those input parameters: Detector name: {fardet | neardet}, Dataset name, Days ago, Release
There are different submission scripts for both detectors, because the fcl files are different and the jobsub command includes additional output data tiers.
Once the reconstruction cron jobs are stable, an unified submission script will be built
This script will submit the jobs to the grid using jobsubs_tools.
Reconstruction as cron jobs¶
Reconstruction is setup as a daily cron job. It will process Far and Near Detector files for NuMI stream with S15-02-05 release.
Reconstruction will start at 02:00 p.m.
Crontab list output (novagpvm01.fnal.gov)
00 14 * * * source /nova/app/home/novapro/KeepUp_oog/Reconstruction_KeepUp/cron_KeepUp.sh fardet 0 S15-02-05
00 16 * * * source /nova/app/home/novapro/KeepUp_oog/Reconstruction_KeepUp/cron_KeepUp.sh neardet 0 S15-02-05
Notes¶
- Since November 2014, Near Detector Raw2Root KeepUp Processing is done under FA14-10-03. Due to an error in geometry detector, files from run number = 10377 up to 10620 were reprocessed. A set of raw files have two children files under different releases.
- Since December 12th, 2014, new dimensions for Raw2Root datasets were defined. After an error reported for NOvA production, dimensions were fixed.
Reconstruction Backprocessing¶
A reconstruction backprocessing request comes usually from the NOvA experiment when a new release with modifications for reconstruction stage is tagged.
Dimensions for datasets that will be used as input of this stage:
file_type = 'importedDetector' and
data_tier = 'artdaq' and
Online.Detector = '${DET}' and
Online.Partition = 1 and
Online.SubRunStartTime >= '${START_DAY_SECONDS}' and
Online.SubRunStartTime <= '${END_DAY_SECONDS}' and
Online.Stream != 0 and Online.Stream != 2 and not
isparentof: ( data_tier = 'reco' and
reconstructed.base_release '${RELEASE}')
keepup_fd_preshutdown_postshutdown.sh | keepup_nd_preshutdown_postshutdown.sh
This scripts will submit the jobs to the grid to run reconstruction keepup.
Renaming wrong named files¶
Due to a DAQ bug, a small set of raw files were wrong named. Those files ended with this name pattern: "_[detector_name]%.raw%.raw_"
Following instructions from Andrew Norman, those files were retired from SAM and new files with the right name were declared. Those files were also renamed on tape (novasamgpvm01.fnal.gov).
Please find attached the list of renamed files.
How to run NOvA interactively?¶
There are two ways of processing a file interactively:
Option 1
This is the preferred option.
[opos@offline-operations-03 ~]$ ssh novagpvm03 -l novapro
Last login: Fri Aug 28 14:51:14 2015 from 131.225.154.193
NOTICE TO USERS
This is a Federal computer (and/or it is directly connected to a
Fermilab local network system) that is the property of the United
States Government. It is for authorized use only. Users (autho-
rized or unauthorized) have no explicit or implicit expectation
of privacy.
Any or all uses of this system and all files on this system may
be intercepted, monitored, recorded, copied, audited, inspected,
and disclosed to authorized site, Department of Energy and law
enforcement personnel, as well as authorized officials of other
agencies, both domestic and foreign. By using this system, the
user consents to such interception, monitoring, recording, copy-
ing, auditing, inspection, and disclosure at the discretion of
authorized site or Department of Energy personnel.
Unauthorized or improper use of this system may result in admin-
istrative disciplinary action and civil and criminal penalties.
By continuing to use this system you indicate your awareness of
and consent to these terms and conditions of use. LOG OFF IMME-
DIATELY if you do not agree to the conditions stated in this
warning.
Fermilab policy and rules for computing, including appropriate
use, may be found at http://www.fnal.gov/cd/main/cpolicy.html
[novapro@novagpvm03 ~]$ setup_nova -r S15-09-28
Release: S15-09-28
Build: debug
PWD: /nova/app/home/novapro
[novapro@novagpvm03 ~]$ samweb get-file-access-url neardet_r00011187_s08_DDcalmu_S15-08-12_v1_data.artdaq.root --schema=xroot
xroot://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/nova/production/raw2root/S15-08-12/neardet/000111/11187/all/neardet_r00011187_s08_DDcalmu_S15-08-12_v1_data.artdaq.root
[novapro@novagpvm03 ~]$nova -c /nova/app/home/novasoft/slf6/novasoft/releases/S15-09-28/Production/fcl/prod_pclist_removebeamspills_job.fcl xroot://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/nova/production/raw2root/S15-08-12/neardet/000111/11187/all/neardet_r00011187_s08_DDcalmu_S15-08-12_v1_data.artdaq.root
Option 2
This option is not recommended as it involves fetching a file to an area where the regular user has no privileges, thus cannot be cleaned (/var/tmp/). Eventually, this area would get full and that would in turn affect production processes as keepup (proxies cannot be copied in).
Example of running a ND calibration file interactively:
[opos@offline-operations-03 ~]$ ssh novagpvm03 -l novapro
Last login: Fri Aug 28 14:51:14 2015 from 131.225.154.193
NOTICE TO USERS
This is a Federal computer (and/or it is directly connected to a
Fermilab local network system) that is the property of the United
States Government. It is for authorized use only. Users (autho-
rized or unauthorized) have no explicit or implicit expectation
of privacy.
Any or all uses of this system and all files on this system may
be intercepted, monitored, recorded, copied, audited, inspected,
and disclosed to authorized site, Department of Energy and law
enforcement personnel, as well as authorized officials of other
agencies, both domestic and foreign. By using this system, the
user consents to such interception, monitoring, recording, copy-
ing, auditing, inspection, and disclosure at the discretion of
authorized site or Department of Energy personnel.
Unauthorized or improper use of this system may result in admin-
istrative disciplinary action and civil and criminal penalties.
By continuing to use this system you indicate your awareness of
and consent to these terms and conditions of use. LOG OFF IMME-
DIATELY if you do not agree to the conditions stated in this
warning.
Fermilab policy and rules for computing, including appropriate
use, may be found at http://www.fnal.gov/cd/main/cpolicy.html
[novapro@novagpvm03 ~]$ setup_nova -r S15-09-28
Release: S15-09-28
Build: debug
PWD: /nova/app/home/novapro
[novapro@novagpvm03 ~]$ samweb locate-file neardet_r00011187_s08_DDcalmu_S15-08-12_v1_data.artdaq.root
enstore:/pnfs/nova/production/raw2root/S15-08-12/neardet/000111/11187/all(694@vpo422)
[novapro@novagpvm03 ~]$ ifdh fetchInput /pnfs/nova/production/raw2root/S15-08-12/neardet/000111/11187/all/neardet_r00011187_s08_DDcalmu_S15-08-12_v1_data.artdaq.root
IFDH_DEBUG=0 => 0
/var/tmp/ifdh_42417_29141/neardet_r00011187_s08_DDcalmu_S15-08-12_v1_data.artdaq.root
[novapro@novagpvm03 ~]$ nova -c /nova/app/home/novasoft/slf6/novasoft/releases/S15-09-28/Production/
.svn/ CMakeLists.txt GNUmakefile README cron/ fcl/ grid/ scripts/ submission/ utilities/
[novapro@novagpvm03 ~]$ nova -c /nova/app/home/novasoft/slf6/novasoft/releases/S15-09-28/Production/fcl/prod_pclist_removebeamspills_job.fcl /var/tmp/ifdh_42417_29141/neardet_r00011187_s08_DDcalmu_S15-08-12_v1_data.artdaq.root
Working Notes on Requests¶
Click in the tittle of the present section to go to the working notes written by OPOS operators on requests made to this group. This is a way to document currently being tested. It started on October 14th 2015. For more details, please reach the OPOS email (opos@fnal.gov).