For all the processing stages of NOvA Offline Production (Keepup), there are input and output dataset defined by SAM dimensions. In order to follow a standard, we suggest to use this pattern for input or output datasets:

prod_[data_tier]_[nova_release]_[detector_id]_[stream]_keepup.[additional_data]

For example:

Raw2Root input dataset: prod_raw_FA14-10-03_nd_others_keepup.2015-04-26
Reconstruction input dataset: prod_artdaq_S15-02-05_nd_numi_keepup.2015-04-18
Reconstruction output dataset: prod_reco_S15-02-05_nd_numi_keepup.r10496_r10798

Raw2Root KeepUp¶

Raw2Root is the first stage for NOvA Offline Production. It will take raw data to produce ROOT files. The output files for this stage will belong to artdaq tier.

SAM Dataset dimensions¶

Dimensions for datasets (draining dataset) that will be used as input of this stage:

file_type       = 'importedDetector' and 
data_tier       = 'raw'  and 
not isparentof: ( data_tier = 'artdaq' and 
DAQ2RawDigit.base_release '${RELEASE}' ) and 
Online.Detector = '${DET}' and 
Online.SubRunEndTime >= '${START_DAY_SECONDS}' and 
Online.SubRunEndTime <= '${END_DAY_SECONDS}' and 
$STREAMDIM and 
Online.TotalEvents > '0' and
file_size < '1288490189' and
Online.SubRunEndTime > '1432098000' 
minus NOVA.ProductionSkip true

The value of

$STREAMDIM

will depend on the specified stream. The stream options are Numi, Cosmics and Others.

Stream	$STREAMDIM
Numi	Online.Stream = 0
Cosmics	Online.Stream = 2
Others	Online.Stream != 0 and Online.Stream != 2 and Online.Stream != 4

Raw2Root KeepUp submission¶

Raw2Root has two steps: input dataset generation and job submission. Raw files are collected every day in the NOvA Detectors, Raw2Root needs to look for the files that were created in the last previous days and process them. The script Raw2RootKeepUp.sh will look for the files created up to 1 month ago iterating over odd dates. A maximum of 16 datasets will be created per Raw2Root running.

Raw2RootKeepUp.sh
Usage: 
$ Raw2RootKeepUp.sh <arguments>

Arguments¶

You must specify the name of the detector which the files belong to, the name of the stream and the NOvA release you want to use to process those files.

  --det DETECTORNAME     Detector name: fardet or neardet
  --stream STREAM        SAM dataset definition to run over
  --release NOVARELEASE  NOvA Release under Raw2Root will process files

Raw2Root KeepUp as cron jobs¶

Raw2Root Keepup involves a set of cron jobs running daily. A single cron job will process one detector and one stream: NuMI, Cosmics or Others. It will start with Far Detector files, processing NuMI, Cosmics and Others streams separately with S14-08-19 release. It will continue with Near Detector files, following the same order for streams with FA14-10-03 release.There are 6 cron jobs placed in a crontab to execute Raw2Root KeepUp.

Raw2Root processing will start at 01:00 a.m.

Crontab list output (novapro@novagpvm01.fnal.gov)

00 01 * * * /nova/app/home/novapro/KeepUp_opos/Raw2Root_KeepUp/Raw2RootKeepUp.sh --det fardet --stream numi --release S15-03-11
00 03 * * * /nova/app/home/novapro/KeepUp_opos/Raw2Root_KeepUp/Raw2RootKeepUp.sh --det fardet --stream cosmics --release S15-03-11
00 05 * * * /nova/app/home/novapro/KeepUp_opos/Raw2Root_KeepUp/Raw2RootKeepUp.sh --det fardet --stream others --release S15-03-11
00 07 * * * /nova/app/home/novapro/KeepUp_opos/Raw2Root_KeepUp/Raw2RootKeepUp.sh --det neardet --stream numi --release FA14-10-03
00 09 * * * /nova/app/home/novapro/KeepUp_opos/Raw2Root_KeepUp/Raw2RootKeepUp.sh --det neardet --stream cosmics --release FA14-10-03
00 11 * * * /nova/app/home/novapro/KeepUp_opos/Raw2Root_KeepUp/Raw2RootKeepUp.sh --det neardet --stream others --release FA14-10-03

Notes¶

Since November 2014, Near Detector Raw2Root KeepUp Processing is done under FA14-10-03. Due to an error in geometry detector, files from run number = 10377 up to 10620 were reprocessed. A set of raw files have two children files under different releases.
Since December 12th, 2014, new dimensions for Raw2Root input datasets were defined. After an error reported for NOvA production, dimensions were fixed.
Since April, 2015, new dimensions for Raw2Root input datasets were defined. After a request from NOvA production, SubRunStartTime metadata field was replaced by SubRunEndTime.
Since end of May, 2015, all files with size greater than 1.2 GB are not being R2R as requested by the collaboration.
Since end of May, 2015, Far Detector Raw2Root KeepUp Processing is done under S15-03-11. A set of raw files have two children files under different releases.

Raw2Root Backprocessing¶

When a Raw2Root Backprocessing request comes from NOvA experiment, the input dataset is created using the NOvA Editor.

Reconstruction Keepup¶

Reconstruction stage takes files from the artdaq tier and will produce output files in reco, caf tiers.

SAM Dataset dimensions¶

Dimensions for datasets (draining dataset) that will be used as input of this stage:

file_type       = 'importedDetector' and
data_tier       = 'artdaq'  and    
Online.Detector = '${DET}' and
Online.Partition = 1 and 
Online.SubRunStartTime >= '${START_DAY_SECONDS}' and
Online.SubRunStartTime <= '${END_DAY_SECONDS}' and
Online.Stream = $STREAM and not
isparentof:     ( data_tier = 'reco')

The list of values for $STREAM can be checked here: http://cdcvs.fnal.gov/lxr/nova/source/Online/pkgs/DAQDataFormats/cxx/include/TriggerDefines.h#241

Scripts¶

Reconstruction scripts are located in: /nova/app/home/novapro/KeepUp_oog/Reconstruction_KeepUp

cron_KeepUp.sh
This script has those input parameters: Detector name: {fardet | neardet}, Stream: {0 | 2 | -1 }, Release
This script will export variables and execute the dataset generation script for Reconstruction KeepUp.

ReconstructionKeepUp.sh
This script has those input parameters: Detector name: {fardet | neardet}, Stream: {0 | 2 | -1 }, Release
This script will setup the environment and build the datasets using the dimensions defined before.
Once the datasets are ready, this script will execute the submission script submitJob_keepup.sh.

submitJob_fd_keepup.sh | submitJob_nd_keepup.sh
This script has those input parameters: Detector name: {fardet | neardet}, Dataset name, Days ago, Release 
There are different submission scripts for both detectors, because the fcl files are different and the jobsub command includes additional output data tiers.
Once the reconstruction cron jobs are stable, an unified submission script will be built
This script will submit the jobs to the grid using jobsubs_tools.

Reconstruction as cron jobs¶

Reconstruction is setup as a daily cron job. It will process Far and Near Detector files for NuMI stream with S15-02-05 release.

Reconstruction will start at 02:00 p.m.

Crontab list output (novagpvm01.fnal.gov)

00 14 * * * source /nova/app/home/novapro/KeepUp_oog/Reconstruction_KeepUp/cron_KeepUp.sh fardet 0 S15-02-05
00 16 * * * source /nova/app/home/novapro/KeepUp_oog/Reconstruction_KeepUp/cron_KeepUp.sh neardet 0 S15-02-05

Notes¶

Since November 2014, Near Detector Raw2Root KeepUp Processing is done under FA14-10-03. Due to an error in geometry detector, files from run number = 10377 up to 10620 were reprocessed. A set of raw files have two children files under different releases.
Since December 12th, 2014, new dimensions for Raw2Root datasets were defined. After an error reported for NOvA production, dimensions were fixed.

Reconstruction Backprocessing¶

A reconstruction backprocessing request comes usually from the NOvA experiment when a new release with modifications for reconstruction stage is tagged.

Dimensions for datasets that will be used as input of this stage:

file_type       = 'importedDetector' and
data_tier       = 'artdaq'  and 
Online.Detector = '${DET}' and
Online.Partition = 1 and 
Online.SubRunStartTime >= '${START_DAY_SECONDS}' and
Online.SubRunStartTime <= '${END_DAY_SECONDS}' and
Online.Stream != 0 and Online.Stream != 2 and not
isparentof:     ( data_tier = 'reco' and 
reconstructed.base_release '${RELEASE}')

keepup_fd_preshutdown_postshutdown.sh | keepup_nd_preshutdown_postshutdown.sh
This scripts will submit the jobs to the grid to run reconstruction keepup.

Renaming wrong named files¶

Due to a DAQ bug, a small set of raw files were wrong named. Those files ended with this name pattern: "_[detector_name]%.raw%.raw_"

Following instructions from Andrew Norman, those files were retired from SAM and new files with the right name were declared. Those files were also renamed on tape (novasamgpvm01.fnal.gov).

Please find attached the list of renamed files.

How to run NOvA interactively?¶

There are two ways of processing a file interactively:

Option 1
This is the preferred option.


[opos@offline-operations-03 ~]$ ssh novagpvm03 -l novapro
Last login: Fri Aug 28 14:51:14 2015 from 131.225.154.193
                              NOTICE TO USERS

       This  is a Federal computer (and/or it is directly connected to a
       Fermilab local network system) that is the property of the United
       States Government.  It is for authorized use only.  Users (autho-
       rized or unauthorized) have no explicit or  implicit  expectation
       of privacy.

       Any  or  all uses of this system and all files on this system may
       be intercepted, monitored, recorded,  copied, audited, inspected,
       and  disclosed  to authorized site, Department of Energy  and law
       enforcement personnel, as  well as authorized officials of  other
       agencies,  both  domestic and foreign.  By using this system, the
       user consents to such interception, monitoring, recording,  copy-
       ing,  auditing,  inspection,  and disclosure at the discretion of
       authorized site or Department of Energy personnel.

       Unauthorized or improper use of this system may result in  admin-
       istrative  disciplinary  action and civil and criminal penalties.
       By continuing to use this system you indicate your  awareness  of
       and  consent to these terms and conditions of use.  LOG OFF IMME-
       DIATELY if you do not agree to  the  conditions  stated  in  this
       warning.

       Fermilab  policy  and  rules for computing, including appropriate
       use, may be found at http://www.fnal.gov/cd/main/cpolicy.html
[novapro@novagpvm03 ~]$ setup_nova -r S15-09-28

Release: S15-09-28
Build:   debug 

PWD: /nova/app/home/novapro
[novapro@novagpvm03 ~]$ samweb get-file-access-url neardet_r00011187_s08_DDcalmu_S15-08-12_v1_data.artdaq.root --schema=xroot
xroot://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/nova/production/raw2root/S15-08-12/neardet/000111/11187/all/neardet_r00011187_s08_DDcalmu_S15-08-12_v1_data.artdaq.root
[novapro@novagpvm03 ~]$nova -c /nova/app/home/novasoft/slf6/novasoft/releases/S15-09-28/Production/fcl/prod_pclist_removebeamspills_job.fcl xroot://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/nova/production/raw2root/S15-08-12/neardet/000111/11187/all/neardet_r00011187_s08_DDcalmu_S15-08-12_v1_data.artdaq.root

Option 2
This option is not recommended as it involves fetching a file to an area where the regular user has no privileges, thus cannot be cleaned (/var/tmp/). Eventually, this area would get full and that would in turn affect production processes as keepup (proxies cannot be copied in).

Example of running a ND calibration file interactively:


[opos@offline-operations-03 ~]$ ssh novagpvm03 -l novapro
Last login: Fri Aug 28 14:51:14 2015 from 131.225.154.193
                              NOTICE TO USERS

       This  is a Federal computer (and/or it is directly connected to a
       Fermilab local network system) that is the property of the United
       States Government.  It is for authorized use only.  Users (autho-
       rized or unauthorized) have no explicit or  implicit  expectation
       of privacy.

       Any  or  all uses of this system and all files on this system may
       be intercepted, monitored, recorded,  copied, audited, inspected,
       and  disclosed  to authorized site, Department of Energy  and law
       enforcement personnel, as  well as authorized officials of  other
       agencies,  both  domestic and foreign.  By using this system, the
       user consents to such interception, monitoring, recording,  copy-
       ing,  auditing,  inspection,  and disclosure at the discretion of
       authorized site or Department of Energy personnel.

       Unauthorized or improper use of this system may result in  admin-
       istrative  disciplinary  action and civil and criminal penalties.
       By continuing to use this system you indicate your  awareness  of
       and  consent to these terms and conditions of use.  LOG OFF IMME-
       DIATELY if you do not agree to  the  conditions  stated  in  this
       warning.

       Fermilab  policy  and  rules for computing, including appropriate
       use, may be found at http://www.fnal.gov/cd/main/cpolicy.html
[novapro@novagpvm03 ~]$ setup_nova -r S15-09-28

Release: S15-09-28
Build:   debug 

PWD: /nova/app/home/novapro
[novapro@novagpvm03 ~]$ samweb locate-file neardet_r00011187_s08_DDcalmu_S15-08-12_v1_data.artdaq.root
enstore:/pnfs/nova/production/raw2root/S15-08-12/neardet/000111/11187/all(694@vpo422)
[novapro@novagpvm03 ~]$ ifdh fetchInput /pnfs/nova/production/raw2root/S15-08-12/neardet/000111/11187/all/neardet_r00011187_s08_DDcalmu_S15-08-12_v1_data.artdaq.root
IFDH_DEBUG=0 => 0
/var/tmp/ifdh_42417_29141/neardet_r00011187_s08_DDcalmu_S15-08-12_v1_data.artdaq.root
[novapro@novagpvm03 ~]$ nova -c /nova/app/home/novasoft/slf6/novasoft/releases/S15-09-28/Production/
.svn/           CMakeLists.txt  GNUmakefile     README          cron/           fcl/            grid/           scripts/        submission/     utilities/
[novapro@novagpvm03 ~]$ nova -c /nova/app/home/novasoft/slf6/novasoft/releases/S15-09-28/Production/fcl/prod_pclist_removebeamspills_job.fcl /var/tmp/ifdh_42417_29141/neardet_r00011187_s08_DDcalmu_S15-08-12_v1_data.artdaq.root

Working Notes on Requests ¶

Click in the tittle of the present section to go to the working notes written by OPOS operators on requests made to this group. This is a way to document currently being tested. It started on October 14th 2015. For more details, please reach the OPOS email (opos@fnal.gov).

Files

FDRenamedFiles.txt (12.6 KB) FDRenamedFiles.txt		Jeny Teheran, 01/30/2015 11:00 AM
NDRenamedFiles.txt (896 Bytes) NDRenamedFiles.txt		Jeny Teheran, 01/30/2015 11:00 AM
NOvASAMDataTiers.png (473 KB) NOvASAMDataTiers.png		Jeny Teheran, 04/27/2015 07:21 PM

Project

General

Profile

Offline Production Operations Service

Wiki

NOvA¶

AWS tests¶

Offsite Probe tests¶

NOvA Period and Epoch Naming¶

Production Systems Health Check¶

Relevant Monitoring Links¶

Data Tiers¶

Naming datasets¶

Raw2Root KeepUp¶

SAM Dataset dimensions¶

Raw2Root KeepUp submission¶

Arguments¶

Raw2Root KeepUp as cron jobs¶

Notes¶

Raw2Root Backprocessing¶

Reconstruction Keepup¶

SAM Dataset dimensions¶

Scripts¶

Reconstruction as cron jobs¶

Notes¶

Reconstruction Backprocessing¶

Renaming wrong named files¶

How to run NOvA interactively?¶

Working Notes on Requests ¶

Project

General

Profile

Offline Production Operations Service

Wiki

NOvA¶

AWS tests¶

Offsite Probe tests¶

NOvA Period and Epoch Naming¶

Production Systems Health Check¶

Relevant Monitoring Links¶

Data Tiers¶

Naming datasets¶

Raw2Root KeepUp¶

SAM Dataset dimensions¶

Raw2Root KeepUp submission¶

Arguments¶

Raw2Root KeepUp as cron jobs¶

Notes¶

Raw2Root Backprocessing¶

Reconstruction Keepup¶

SAM Dataset dimensions¶

Scripts¶

Reconstruction as cron jobs¶

Notes¶

Reconstruction Backprocessing¶

Renaming wrong named files¶

How to run NOvA interactively?¶

Working Notes on Requests¶

Working Notes on Requests ¶