NOvA » NOvA-ART

h1. Archiving data and access to data on tapes

h2. File Sets Pending Archival

List of the file needed to be archived:

h3. 2.  /nova/data/mc/.ToBeArchived  ~ pending offline permission, set to be obtained November 3rd, 2014

For some of these files, this location is declared to SAM and that location also needs to be deleted. Directories that include files like this are: 

   S12-11-16 (Almost all files already on tape; these don't need to move, just delete file and SAM location)

   S13-01-15 (A few of these files are already on tape; their blue arc location ISN'T recorded, so for this handful, can just delete)

   S13-07-22 (These files seem to all be declared to SAM, but only have the bluearc location. So these need the full treatment - they need to be archived and new file location told to SAM, and the current file location needs to be removed)

   S13-10-11 (These files seem to all be declared to SAM, but only have the bluearc location. So these need the full treatment - they need to be archived and new file location told to SAM, and the current file location needs to be removed)

   S13-12-13 (These files are already on tape; these don't need to move, just delete file and SAM location)

h3. 3.  /nova/prod/mc/.ToBeArchived  ~ pending offline permission, set to be obtained November 3rd, 2014

h2. Status page for the FTS 

http://novasamgpvm01.fnal.gov:8888/fts/status

h1. File Transfer details (BlueArc to dCache)

Files on bluearc which is no longer in use, needs to be archived on pnfs. The first such file which is moved to pnfs is /nova/prod/mc/S13-06-05/cosmics/fd/ which is about 6.7 TB. The pnfs location of this file is /pnfs/nova/archives/2014-APR/mc/S13-06-05/cosmics/fd/xx where xx is the file number. The files on bluearc were divided into further smaller directories according to their run numbers. The sub-directory with fewer files make file operation much more efficient than putting all files together in one directory. The above file from bluearc were divided into a list of 100 sub-directories containing list of 111 files each.

h3. Copy, Verify and Remove

To do operations like copy, verify and remove a script is written which runs by putting arguments copy, verify and remove in the command line. The files to be moved from bluearc needs to be checked with pnfs whether it already exist there or not. If it already exists then its copying is skipped and copy starts for the next file. 

After copying the file to the pnfs it is verified by checking the size of the file. When the verification is done it is removed from the bluearc. The following steps are involved while doing file operations:

1. Login to any VMs (novagpvm's)

2. Go to the directory where script is written and use the following commands to get access to file operation as 'novapro'

2. ksu novapro (This gives one permission to do operations with files)

3. source novaprosource

After these steps one is ready to run scripts for file operations.

h2. Details of files move to dCache off the bluearc

h3. S13-06-05/cosmics/fd ~ 6.7 TB

The file on Bluearc (/nova/prod/mc/S13-06-05/cosmics/fd) ~6.7 TB have been moved to dCache (/pnfs/nova/archives/2014-APR/mc/S13-06-05/cosmics/fd/xx) where xx is the file number. These files have been divided into smaller parts to make file operation easier.

h3. S14-02-05/cosmics/fd ~ 7.1 TB

Following the similar procedures the files these files are also transferred to dCache (/pnfs/nova/archives/2014-MAY/mc/B13-10-23/cosmics/fd/).

h3. B13-10-23/cosmics/fd ~ 16 TB

Again the files from this area was split into a set of smaller files for quicker operations. Following files were removed from the BlueArc without copying it to the dCache:

/nova/prod/mc/B13-10-23/cosmics/fd/ImprovedTrans_AdamReadoutSim

/nova/prod/mc/B13-10-23/cosmics/fd/ImprovedTrans_multiplexReadoutSim

/nova/prod/mc/B13-10-23/cosmics/fd/ImprovedTrans_newMultiplexReadoutSim

Following files was copied to the dCache first and then it was removed off the BlueArc:

/pnfs/nova/archives/2014-MAY/mc/B13-10-23/cosmics/fd/.

These files were copied to: /pnfs/nova/archives/2014-APR/mc/S13-06-05/cosmics/fd/

This area has directories with run numbers for the ease of access.

h3. S13-10-11/cosmics/fd ~ 5.1 TB

The same strategy was used for this data set as well. This time the files have been kept in the sub folders of the iterations as they were in bluearc.

/nova/prod/mc/S13-10-11/cosmics/fd/Oct10ReadoutSim_ChanMask11342

/nova/prod/mc/S13-10-11/cosmics/fd/Oct2ReadoutSim_ChanMask11342

/nova/prod/mc/S13-10-11/cosmics/fd/StaggerOct10ReadoutSim_ChanMask11342

These files are archives in dCache at:

/pnfs/nova/archives/2014-JULY/mc/S13-10-11/cosmics/fd/

It has three sub-directories:

/pnfs/nova/archives/2014-JULY/mc/S13-10-11/cosmics/fd/Oct10ReadoutSim_ChanMask11342

/pnfs/nova/archives/2014-JULY/mc/S13-10-11/cosmics/fd/Oct2ReadoutSim_ChanMask11342

/pnfs/nova/archives/2014-JULY/mc/S13-10-11/cosmics/fd/StaggerOct10ReadoutSim_ChanMask11342

The folder /pnfs/nova/archives/2014-JULY/mc/S13-10-11/cosmics/fd/Oct10ReadoutSim_ChanMask11342 is empty now because the directories 1 to 10 in /pnfs/nova/archives/2014-JULY/mc/S13-10-11/cosmics/fd/ should have been inside it. Working on to put them back to their correct location and update this space again. The google spread sheet is attached.

h3. S13-06-13/cosmics/fd ~ 5.8 TB

All the files from:

/nova/prod/mc/S13-06-13/cosmics/fd/

have been moved to:

/pnfs/nova/archives/2014-JULY/mc/S13-06-13/cosmics/fd/

The Google spread sheet is attached.

h3. /nova/ana/calibration/FarDet/ ~ (7.1) TB

All the files *pchits and *pchitstop files from /nova/ana/calibration/FarDet/S13-09-04/ are to be transferred to dCache (/pnfs/nova/archives/2014-JULY/mc/calibFiles/S13-09-04/). The files of reference directory is kept in reference directory (/pnfs/nova/archives/2014-JULY/mc/calibFiles/S13-09-04/reference) and the reference.MANGLED files are in reference.MANGLED directory(/pnfs/nova/archives/2014-JULY/mc/calibFiles/S13-09-04/reference.MANGLED). 

Since novapro can not delete these files from current bluearc (/nova/ana/calibration/FarDet/S13-09-04/) location, Gavin have to delete them to get all these free space. The Google spreadsheet is attached.

h3. S12-12-12/genie/{fd, nd} ~ 13TB

The FD and ND (/nova/prod/mc/S12-12-12/genie/) files have been archived to dCache (/pnfs/nova/archives/2014-AUG/mc/S12-12-12/genie/).

h3. mdc_S12-06-17 ~ 2.4TB

The FD and ND (/nova/ana/caf/mdc/) files have been archived to dCache (/pnfs/nova/archives/2014-Oct/mdc/). Few of the files could not be verified for its size dCache. It is copied again in /pnfs/nova/archives/2014-Oct/mdc/uncopied.

h3. /nova/ana/caf/base/ ~ 12 TB

The FD and ND (/nova/ana/caf/base/) files is archived to dCache (/pnfs/nova/archives/2014-Oct/base/)

h3. /nova/ana/MOVED/chadj ~ 114 G (permission from Mark Messier on Oct 13, 2014)

These files have been copied to /pnfs/nova/archives/2014-Oct/MOVED/chadj/ and same directory structure have been maintained. The logs folder and the labview folders were tarred and copied to the /pnfs/nova/archives/2014-Oct/MOVED/chadj/logs and /pnfs/nova/archives/2014-Oct/MOVED/chadj/labview respectively. As these files belong to chadj novapro can't delete it. Andrew Norman has agreed to delete these. Waiting for him to delete.

h3.  /nova/prod/mc/S13-02-26 ~ 12 TB, except NDOS files

It has been archived to /pnfs/nova/archives/2014-Nov/mc/S13-02-26/.

h2. Details about metadata for given production release.

h3. [[S12-10-31]]

h3. [[S12.06.17]]

h3. [[S12.02.14]]

h3. [[S11.11.06]]