Project

General

Profile

NOvA

AWS tests

Here you can find a summary of the AWS tests ran by the OPOS team. Test started being ran by OPOS on January 2016.

Offsite Probe tests

Here you can find details about this activity.

NOvA Period and Epoch Naming

https://cdcvs.fnal.gov/redmine/projects/novaart/wiki/Period_and_Epoch_Naming

Production Systems Health Check

Follow the instructions available here to monitor the NOvA production systems.

Relevant Monitoring Links

To monitor the NOvA Offline Systems, you can use the following links:

NOvA SAM station
http://samweb.fnal.gov:8480/station_monitor/nova/stations/nova/projects

NOvA Production FTS

NOvA Fifemon
http://fifemon.fnal.gov/monitor/experiment/nova

NOvA Fifemon Batch
http://fifemon.fnal.gov/monitor/experiment/nova/batch

NOvA DAQ2RawDigits Base Releases
http://nusoft.fnal.gov/nova/production/sandbox/d3_raw2root_ana.html

You can also check overall monitoring tools in: https://cdcvs.fnal.gov/redmine/projects/offline_production_operations_service/wiki/Hello_Team#Hello-Team

Data Tiers

NOvASAMDataTiers.png

Naming datasets

For all the processing stages of NOvA Offline Production (Keepup), there are input and output dataset defined by SAM dimensions. In order to follow a standard, we suggest to use this pattern for input or output datasets:

prod_[data_tier]_[nova_release]_[detector_id]_[stream]_keepup.[additional_data]

For example:

  • Raw2Root input dataset: prod_raw_FA14-10-03_nd_others_keepup.2015-04-26
  • Reconstruction input dataset: prod_artdaq_S15-02-05_nd_numi_keepup.2015-04-18
  • Reconstruction output dataset: prod_reco_S15-02-05_nd_numi_keepup.r10496_r10798

Raw2Root KeepUp

Raw2Root is the first stage for NOvA Offline Production. It will take raw data to produce ROOT files. The output files for this stage will belong to artdaq tier.

SAM Dataset dimensions

Dimensions for datasets (draining dataset) that will be used as input of this stage:

file_type       = 'importedDetector' and 
data_tier       = 'raw'  and 
not isparentof: ( data_tier = 'artdaq' and 
DAQ2RawDigit.base_release '${RELEASE}' ) and 
Online.Detector = '${DET}' and 
Online.SubRunEndTime >= '${START_DAY_SECONDS}' and 
Online.SubRunEndTime <= '${END_DAY_SECONDS}' and 
$STREAMDIM and 
Online.TotalEvents > '0' and
file_size < '1288490189' and
Online.SubRunEndTime > '1432098000' 
minus NOVA.ProductionSkip true 

The value of

$STREAMDIM
will depend on the specified stream. The stream options are Numi, Cosmics and Others.

Stream $STREAMDIM
Numi Online.Stream = 0
Cosmics Online.Stream = 2
Others Online.Stream != 0 and Online.Stream != 2 and Online.Stream != 4

Raw2Root KeepUp submission

Raw2Root has two steps: input dataset generation and job submission. Raw files are collected every day in the NOvA Detectors, Raw2Root needs to look for the files that were created in the last previous days and process them. The script Raw2RootKeepUp.sh will look for the files created up to 1 month ago iterating over odd dates. A maximum of 16 datasets will be created per Raw2Root running.

Raw2RootKeepUp.sh
Usage:
$ Raw2RootKeepUp.sh <arguments>

Arguments

You must specify the name of the detector which the files belong to, the name of the stream and the NOvA release you want to use to process those files.

  --det DETECTORNAME     Detector name: fardet or neardet
  --stream STREAM        SAM dataset definition to run over
  --release NOVARELEASE  NOvA Release under Raw2Root will process files

Raw2Root KeepUp as cron jobs

Raw2Root Keepup involves a set of cron jobs running daily. A single cron job will process one detector and one stream: NuMI, Cosmics or Others. It will start with Far Detector files, processing NuMI, Cosmics and Others streams separately with S14-08-19 release. It will continue with Near Detector files, following the same order for streams with FA14-10-03 release.There are 6 cron jobs placed in a crontab to execute Raw2Root KeepUp.

Raw2Root processing will start at 01:00 a.m.

Crontab list output ()

00 01 * * * /nova/app/home/novapro/KeepUp_opos/Raw2Root_KeepUp/Raw2RootKeepUp.sh --det fardet --stream numi --release S15-03-11
00 03 * * * /nova/app/home/novapro/KeepUp_opos/Raw2Root_KeepUp/Raw2RootKeepUp.sh --det fardet --stream cosmics --release S15-03-11
00 05 * * * /nova/app/home/novapro/KeepUp_opos/Raw2Root_KeepUp/Raw2RootKeepUp.sh --det fardet --stream others --release S15-03-11
00 07 * * * /nova/app/home/novapro/KeepUp_opos/Raw2Root_KeepUp/Raw2RootKeepUp.sh --det neardet --stream numi --release FA14-10-03
00 09 * * * /nova/app/home/novapro/KeepUp_opos/Raw2Root_KeepUp/Raw2RootKeepUp.sh --det neardet --stream cosmics --release FA14-10-03
00 11 * * * /nova/app/home/novapro/KeepUp_opos/Raw2Root_KeepUp/Raw2RootKeepUp.sh --det neardet --stream others --release FA14-10-03

Notes

  • Since November 2014, Near Detector Raw2Root KeepUp Processing is done under FA14-10-03. Due to an error in geometry detector, files from run number = 10377 up to 10620 were reprocessed. A set of raw files have two children files under different releases.
  • Since December 12th, 2014, new dimensions for Raw2Root input datasets were defined. After an error reported for NOvA production, dimensions were fixed.
  • Since April, 2015, new dimensions for Raw2Root input datasets were defined. After a request from NOvA production, SubRunStartTime metadata field was replaced by SubRunEndTime.
  • Since end of May, 2015, all files with size greater than 1.2 GB are not being R2R as requested by the collaboration.
  • Since end of May, 2015, Far Detector Raw2Root KeepUp Processing is done under S15-03-11. A set of raw files have two children files under different releases.

Raw2Root Backprocessing

When a Raw2Root Backprocessing request comes from NOvA experiment, the input dataset is created using the NOvA Editor.

Reconstruction Keepup

Reconstruction stage takes files from the artdaq tier and will produce output files in reco, caf tiers.

SAM Dataset dimensions

Dimensions for datasets (draining dataset) that will be used as input of this stage:

file_type       = 'importedDetector' and
data_tier       = 'artdaq'  and    
Online.Detector = '${DET}' and
Online.Partition = 1 and 
Online.SubRunStartTime >= '${START_DAY_SECONDS}' and
Online.SubRunStartTime <= '${END_DAY_SECONDS}' and
Online.Stream = $STREAM and not
isparentof:     ( data_tier = 'reco')

The list of values for $STREAM can be checked here: http://cdcvs.fnal.gov/lxr/nova/source/Online/pkgs/DAQDataFormats/cxx/include/TriggerDefines.h#241

Scripts

Reconstruction scripts are located in: /nova/app/home/novapro/KeepUp_oog/Reconstruction_KeepUp

cron_KeepUp.sh
This script has those input parameters: Detector name: {fardet | neardet}, Stream: {0 | 2 | -1 }, Release
This script will export variables and execute the dataset generation script for Reconstruction KeepUp.
ReconstructionKeepUp.sh
This script has those input parameters: Detector name: {fardet | neardet}, Stream: {0 | 2 | -1 }, Release
This script will setup the environment and build the datasets using the dimensions defined before.
Once the datasets are ready, this script will execute the submission script submitJob_keepup.sh.
submitJob_fd_keepup.sh | submitJob_nd_keepup.sh
This script has those input parameters: Detector name: {fardet | neardet}, Dataset name, Days ago, Release
There are different submission scripts for both detectors, because the fcl files are different and the jobsub command includes additional output data tiers.
Once the reconstruction cron jobs are stable, an unified submission script will be built
This script will submit the jobs to the grid using jobsubs_tools.

Reconstruction as cron jobs

Reconstruction is setup as a daily cron job. It will process Far and Near Detector files for NuMI stream with S15-02-05 release.

Reconstruction will start at 02:00 p.m.

Crontab list output (novagpvm01.fnal.gov)

00 14 * * * source /nova/app/home/novapro/KeepUp_oog/Reconstruction_KeepUp/cron_KeepUp.sh fardet 0 S15-02-05
00 16 * * * source /nova/app/home/novapro/KeepUp_oog/Reconstruction_KeepUp/cron_KeepUp.sh neardet 0 S15-02-05

Notes

  • Since November 2014, Near Detector Raw2Root KeepUp Processing is done under FA14-10-03. Due to an error in geometry detector, files from run number = 10377 up to 10620 were reprocessed. A set of raw files have two children files under different releases.
  • Since December 12th, 2014, new dimensions for Raw2Root datasets were defined. After an error reported for NOvA production, dimensions were fixed.

Reconstruction Backprocessing

A reconstruction backprocessing request comes usually from the NOvA experiment when a new release with modifications for reconstruction stage is tagged.

Dimensions for datasets that will be used as input of this stage:

file_type       = 'importedDetector' and
data_tier       = 'artdaq'  and 
Online.Detector = '${DET}' and
Online.Partition = 1 and 
Online.SubRunStartTime >= '${START_DAY_SECONDS}' and
Online.SubRunStartTime <= '${END_DAY_SECONDS}' and
Online.Stream != 0 and Online.Stream != 2 and not
isparentof:     ( data_tier = 'reco' and 
reconstructed.base_release '${RELEASE}')

keepup_fd_preshutdown_postshutdown.sh | keepup_nd_preshutdown_postshutdown.sh
This scripts will submit the jobs to the grid to run reconstruction keepup.

Renaming wrong named files

Due to a DAQ bug, a small set of raw files were wrong named. Those files ended with this name pattern: "_[detector_name]%.raw%.raw_"

Following instructions from Andrew Norman, those files were retired from SAM and new files with the right name were declared. Those files were also renamed on tape (novasamgpvm01.fnal.gov).

Please find attached the list of renamed files.

How to run NOvA interactively?

There are two ways of processing a file interactively:

Option 1
This is the preferred option.


[opos@offline-operations-03 ~]$ ssh novagpvm03 -l novapro
Last login: Fri Aug 28 14:51:14 2015 from 131.225.154.193
                              NOTICE TO USERS

       This  is a Federal computer (and/or it is directly connected to a
       Fermilab local network system) that is the property of the United
       States Government.  It is for authorized use only.  Users (autho-
       rized or unauthorized) have no explicit or  implicit  expectation
       of privacy.

       Any  or  all uses of this system and all files on this system may
       be intercepted, monitored, recorded,  copied, audited, inspected,
       and  disclosed  to authorized site, Department of Energy  and law
       enforcement personnel, as  well as authorized officials of  other
       agencies,  both  domestic and foreign.  By using this system, the
       user consents to such interception, monitoring, recording,  copy-
       ing,  auditing,  inspection,  and disclosure at the discretion of
       authorized site or Department of Energy personnel.

       Unauthorized or improper use of this system may result in  admin-
       istrative  disciplinary  action and civil and criminal penalties.
       By continuing to use this system you indicate your  awareness  of
       and  consent to these terms and conditions of use.  LOG OFF IMME-
       DIATELY if you do not agree to  the  conditions  stated  in  this
       warning.

       Fermilab  policy  and  rules for computing, including appropriate
       use, may be found at http://www.fnal.gov/cd/main/cpolicy.html
[novapro@novagpvm03 ~]$ setup_nova -r S15-09-28

Release: S15-09-28
Build:   debug 

PWD: /nova/app/home/novapro
[novapro@novagpvm03 ~]$ samweb get-file-access-url neardet_r00011187_s08_DDcalmu_S15-08-12_v1_data.artdaq.root --schema=xroot
xroot://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/nova/production/raw2root/S15-08-12/neardet/000111/11187/all/neardet_r00011187_s08_DDcalmu_S15-08-12_v1_data.artdaq.root
[novapro@novagpvm03 ~]$nova -c /nova/app/home/novasoft/slf6/novasoft/releases/S15-09-28/Production/fcl/prod_pclist_removebeamspills_job.fcl xroot://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/nova/production/raw2root/S15-08-12/neardet/000111/11187/all/neardet_r00011187_s08_DDcalmu_S15-08-12_v1_data.artdaq.root

Option 2
This option is not recommended as it involves fetching a file to an area where the regular user has no privileges, thus cannot be cleaned (/var/tmp/). Eventually, this area would get full and that would in turn affect production processes as keepup (proxies cannot be copied in).

Example of running a ND calibration file interactively:


[opos@offline-operations-03 ~]$ ssh novagpvm03 -l novapro
Last login: Fri Aug 28 14:51:14 2015 from 131.225.154.193
                              NOTICE TO USERS

       This  is a Federal computer (and/or it is directly connected to a
       Fermilab local network system) that is the property of the United
       States Government.  It is for authorized use only.  Users (autho-
       rized or unauthorized) have no explicit or  implicit  expectation
       of privacy.

       Any  or  all uses of this system and all files on this system may
       be intercepted, monitored, recorded,  copied, audited, inspected,
       and  disclosed  to authorized site, Department of Energy  and law
       enforcement personnel, as  well as authorized officials of  other
       agencies,  both  domestic and foreign.  By using this system, the
       user consents to such interception, monitoring, recording,  copy-
       ing,  auditing,  inspection,  and disclosure at the discretion of
       authorized site or Department of Energy personnel.

       Unauthorized or improper use of this system may result in  admin-
       istrative  disciplinary  action and civil and criminal penalties.
       By continuing to use this system you indicate your  awareness  of
       and  consent to these terms and conditions of use.  LOG OFF IMME-
       DIATELY if you do not agree to  the  conditions  stated  in  this
       warning.

       Fermilab  policy  and  rules for computing, including appropriate
       use, may be found at http://www.fnal.gov/cd/main/cpolicy.html
[novapro@novagpvm03 ~]$ setup_nova -r S15-09-28

Release: S15-09-28
Build:   debug 

PWD: /nova/app/home/novapro
[novapro@novagpvm03 ~]$ samweb locate-file neardet_r00011187_s08_DDcalmu_S15-08-12_v1_data.artdaq.root
enstore:/pnfs/nova/production/raw2root/S15-08-12/neardet/000111/11187/all(694@vpo422)
[novapro@novagpvm03 ~]$ ifdh fetchInput /pnfs/nova/production/raw2root/S15-08-12/neardet/000111/11187/all/neardet_r00011187_s08_DDcalmu_S15-08-12_v1_data.artdaq.root
IFDH_DEBUG=0 => 0
/var/tmp/ifdh_42417_29141/neardet_r00011187_s08_DDcalmu_S15-08-12_v1_data.artdaq.root
[novapro@novagpvm03 ~]$ nova -c /nova/app/home/novasoft/slf6/novasoft/releases/S15-09-28/Production/
.svn/           CMakeLists.txt  GNUmakefile     README          cron/           fcl/            grid/           scripts/        submission/     utilities/
[novapro@novagpvm03 ~]$ nova -c /nova/app/home/novasoft/slf6/novasoft/releases/S15-09-28/Production/fcl/prod_pclist_removebeamspills_job.fcl /var/tmp/ifdh_42417_29141/neardet_r00011187_s08_DDcalmu_S15-08-12_v1_data.artdaq.root

Working Notes on Requests

Click in the tittle of the present section to go to the working notes written by OPOS operators on requests made to this group. This is a way to document currently being tested. It started on October 14th 2015. For more details, please reach the OPOS email ().