NOvA¶

Data Tiers¶

Naming datasets¶

For all the processing stages of NOvA Offline Production (Keepup), there are input and output dataset defined by SAM dimensions. In order to follow a standard, we suggest to use this pattern for input or output datasets:

prod_[data_tier]_[nova_release]_[detector_id]_[stream]_keepup.[additional_data]

For example:

• Raw2Root input dataset: prod_raw_FA14-10-03_nd_others_keepup.2015-04-26
• Reconstruction input dataset: prod_artdaq_S15-02-05_nd_numi_keepup.2015-04-18
• Reconstruction output dataset: prod_reco_S15-02-05_nd_numi_keepup.r10496_r10798

Raw2Root KeepUp¶

Raw2Root is the first stage for NOvA Offline Production. It will take raw data to produce ROOT files. The output files for this stage will belong to artdaq tier.

SAM Dataset dimensions¶

Dimensions for datasets (draining dataset) that will be used as input of this stage:

file_type       = 'importedDetector' and
data_tier       = 'raw'  and not
isparentof:     ( data_tier = 'artdaq' ) and
Online.Detector = '${DET}' and
Online.SubRunEndTime >= '${START_DAY_SECONDS}' and
Online.SubRunEndTime <= '${END_DAY_SECONDS}' and
Online.Stream = $STREAM and
Online.TotalEvents > '0'

Raw2Root KeepUp submission¶

Raw2Root has two steps: input dataset generation and job submission. Raw files are collected every day in the NOvA Detectors, Raw2Root needs to look for the files that were created in the last previous days and process them. The script Raw2RootKeepUp.sh will look for the files created up to 1 month ago iterating over odd dates. A maximum of 16 datasets will be created per Raw2Root running.

Raw2RootKeepUp.sh
Usage: 
$ Raw2RootKeepUp.sh <arguments>

Arguments¶

You must specify the name of the detector which the files belong, the name of the stream and the NOvA release you want to use to process those files.

  --det DETECTORNAME     Detector name: fardet or neardet
  --stream STREAM        SAM dataset definition to run over
  --release NOVARELEASE  NOvA Release under Raw2Root will process files

Raw2Root KeepUp as cron jobs¶

Raw2Root Keepup involves a set of cron jobs running daily. A single cron job will process one detector and one stream: NuMI, Cosmics or Others. It will start with Far Detector files, processing NuMI, Cosmics and Others streams separately with S14-08-19 release. It will continue with Near Detector files, following the same order for streams with FA14-10-03 release.There are 6 cron jobs placed in a crontab to execute Raw2Root KeepUp.

Raw2Root processing will start at 01:00 a.m.

Crontab list output (novagpvm01.fnal.gov)

00 01 * * * /nova/app/home/novapro/KeepUp_opos/Raw2Root_KeepUp/Raw2RootKeepUp.sh --det fardet --stream numi --release S14-08-19
00 03 * * * /nova/app/home/novapro/KeepUp_opos/Raw2Root_KeepUp/Raw2RootKeepUp.sh --det fardet --stream cosmics --release S14-08-19
00 05 * * * /nova/app/home/novapro/KeepUp_opos/Raw2Root_KeepUp/Raw2RootKeepUp.sh --det fardet --stream others --release S14-08-19
00 07 * * * /nova/app/home/novapro/KeepUp_opos/Raw2Root_KeepUp/Raw2RootKeepUp.sh --det neardet --stream numi --release FA14-10-03
00 09 * * * /nova/app/home/novapro/KeepUp_opos/Raw2Root_KeepUp/Raw2RootKeepUp.sh --det neardet --stream cosmics --release FA14-10-03
00 11 * * * /nova/app/home/novapro/KeepUp_opos/Raw2Root_KeepUp/Raw2RootKeepUp.sh --det neardet --stream others --release FA14-10-03

Notes¶

• Since November 2014, Near Detector Raw2Root KeepUp Processing is done under FA14-10-03. Due to an error in geometry detector, files from run number = 10377 up to 10620 were reprocessed. A set of raw files have two children files under different releases.
• Since December 12th, 2014, new dimensions for Raw2Root input datasets were defined. After an error reported for NOvA production, dimensions were fixed.
• Since April, 2015, new dimensions for Raw2Root input datasets were defined. After a request from NOvA production, SubRunStartTime metadata field was replaced by SubRunEndTime.

Raw2Root Backprocessing¶

When a Raw2Root Backprocessing request comes from NOvA experiment, the input dataset is created using the NOvA Editor.

Reconstruction Keepup¶

Reconstruction stage takes files from the artdaq tier and will produce output files in reco, caf tiers.

SAM Dataset dimensions¶

Dimensions for datasets (draining dataset) that will be used as input of this stage:

file_type       = 'importedDetector' and
data_tier       = 'artdaq'  and    
Online.Detector = '${DET}' and
Online.Partition = 1 and 
Online.SubRunStartTime >= '${START_DAY_SECONDS}' and
Online.SubRunStartTime <= '${END_DAY_SECONDS}' and
Online.Stream = $STREAM and not
isparentof:     ( data_tier = 'reco')

Scripts¶

Reconstruction scripts are located in: /nova/app/home/novapro/KeepUp_oog/Reconstruction_KeepUp

cron_KeepUp.sh
This script has those input parameters: Detector name: {fardet | neardet}, Stream: {0 | 2 | -1 }, Release
This script will export variables and execute the dataset generation script for Reconstruction KeepUp.

ReconstructionKeepUp.sh
This script has those input parameters: Detector name: {fardet | neardet}, Stream: {0 | 2 | -1 }, Release
This script will setup the environment and build the datasets using the dimensions defined before.
Once the datasets are ready, this script will execute the submission script submitJob_keepup.sh.

submitJob_fd_keepup.sh | submitJob_nd_keepup.sh
This script has those input parameters: Detector name: {fardet | neardet}, Dataset name, Days ago, Release 
There are different submission scripts for both detectors, because the fcl files are different and the jobsub command includes additional output data tiers.
Once the reconstruction cron jobs are stable, an unified submission script will be built
This script will submit the jobs to the grid using jobsubs_tools.

Reconstruction as cron jobs¶

Reconstruction is setup as a daily cron job. It will process Far and Near Detector files for NuMI stream with S15-02-05 release.

Reconstruction will start at 02:00 p.m.

Crontab list output (novagpvm01.fnal.gov)

00 14 * * * source /nova/app/home/novapro/KeepUp_oog/Reconstruction_KeepUp/cron_KeepUp.sh fardet 0 S15-02-05
00 16 * * * source /nova/app/home/novapro/KeepUp_oog/Reconstruction_KeepUp/cron_KeepUp.sh neardet 0 S15-02-05

Notes¶

• Since November 2014, Near Detector Raw2Root KeepUp Processing is done under FA14-10-03. Due to an error in geometry detector, files from run number = 10377 up to 10620 were reprocessed. A set of raw files have two children files under different releases.
• Since December 12th, 2014, new dimensions for Raw2Root datasets were defined. After an error reported for NOvA production, dimensions were fixed.

Reconstruction Backprocessing¶

A reconstruction backprocessing request comes usually from the NOvA experiment when a new release with modifications for reconstruction stage is tagged.

Dimensions for datasets that will be used as input of this stage:

file_type       = 'importedDetector' and
data_tier       = 'artdaq'  and 
Online.Detector = '${DET}' and
Online.Partition = 1 and 
Online.SubRunStartTime >= '${START_DAY_SECONDS}' and
Online.SubRunStartTime <= '${END_DAY_SECONDS}' and
Online.Stream != 0 and Online.Stream != 2 and not
isparentof:     ( data_tier = 'reco' and 
reconstructed.base_release '${RELEASE}')

keepup_fd_preshutdown_postshutdown.sh | keepup_nd_preshutdown_postshutdown.sh
This scripts will submit the jobs to the grid to run reconstruction keepup.

Renaming wrong named files¶

Due to a DAQ bug, a small set of raw files were wrong named. Those files ended with this name pattern: "_[detector_name]%.raw%.raw_"

Following instructions from Andrew Norman, those files were retired from SAM and new files with the right name were declared. Those files were also renamed on tape (novasamgpvm01.fnal.gov).

Please find attached the list of renamed files.

How to run NOvA interactively?¶

How to fecth files using IFDH¶