Project

General

Profile

Archiving data and access to data on tapes » History » Version 104

Pavanpoot Pandey, 07/27/2015 08:15 PM

1 1 Jarek Nowak
h1. Archiving data and access to data on tapes
2 1 Jarek Nowak
3 53 Susan Lein
h2. File Sets Pending Archival
4 53 Susan Lein
5 54 Pavanpoot Pandey
List of the file needed to be archived:
6 54 Pavanpoot Pandey
7 104 Pavanpoot Pandey
h3. 1.  /nova/prod/: There is recent surge in the occupancy of this area, following data sets have been identified in priority order to be archived:
8 103 Pavanpoot Pandey
9 103 Pavanpoot Pandey
.S12-12-12_cosmic
10 103 Pavanpoot Pandey
None
11 103 Pavanpoot Pandey
S13-01-15
12 103 Pavanpoot Pandey
S13-02-03
13 103 Pavanpoot Pandey
S13-02-26
14 103 Pavanpoot Pandey
S13-06-05
15 103 Pavanpoot Pandey
S13-06-13
16 103 Pavanpoot Pandey
S13-06-18
17 103 Pavanpoot Pandey
S13-06-26
18 103 Pavanpoot Pandey
S13-12-13
19 103 Pavanpoot Pandey
S14-01-20
20 103 Pavanpoot Pandey
S14-02-05
21 103 Pavanpoot Pandey
S14-02-05a
22 103 Pavanpoot Pandey
S14-03-06
23 103 Pavanpoot Pandey
S14-03-25
24 103 Pavanpoot Pandey
S14-05-05
25 103 Pavanpoot Pandey
S14-05-08
26 103 Pavanpoot Pandey
S14-05-12
27 103 Pavanpoot Pandey
S14-07-03
28 103 Pavanpoot Pandey
S14-07-11
29 103 Pavanpoot Pandey
S14-08-15
30 1 Jarek Nowak
S14-08-19
31 1 Jarek Nowak
s14-01-20
32 104 Pavanpoot Pandey
33 104 Pavanpoot Pandey
34 104 Pavanpoot Pandey
35 104 Pavanpoot Pandey
36 104 Pavanpoot Pandey
h3. 2.  /nova/../users/ : These directories belong to the users who have already left the collaboration. These also needs to be archived:
37 104 Pavanpoot Pandey
38 104 Pavanpoot Pandey
182G        /nova/app/users/denis
39 104 Pavanpoot Pandey
178G        /nova/app/users/timdkut
40 104 Pavanpoot Pandey
176G        /nova/app/users/betan009
41 104 Pavanpoot Pandey
2078G        /nova/ana/users/nsmayer
42 104 Pavanpoot Pandey
43 104 Pavanpoot Pandey
44 103 Pavanpoot Pandey
45 103 Pavanpoot Pandey
46 87 Susan Lein
h3. 2.  /nova/data/mc/.ToBeArchived  ~ permission obtained November 3rd, 2014
47 75 Susan Lein
48 76 Susan Lein
For some of these files, this location is declared to SAM and that location also needs to be deleted. Directories that include files like this are: 
49 78 Susan Lein
   S13-01-15 (A few of these files are already on tape; their blue arc location ISN'T recorded, so for this handful, can just delete)
50 79 Susan Lein
   S13-07-22 (These files seem to all be declared to SAM, but only have the bluearc location. So these need the full treatment - they need to be archived and new file location told to SAM, and the current file location needs to be removed)
51 80 Susan Lein
   S13-10-11 (These files seem to all be declared to SAM, but only have the bluearc location. So these need the full treatment - they need to be archived and new file location told to SAM, and the current file location needs to be removed)
52 81 Susan Lein
   S13-12-13 (These files are already on tape; these don't need to move, just delete file and SAM location)
53 76 Susan Lein
54 75 Susan Lein
h3. 3.  /nova/prod/mc/.ToBeArchived  ~ pending offline permission, set to be obtained November 3rd, 2014
55 69 Pavanpoot Pandey
56 1 Jarek Nowak
h2. Status page for the FTS 
57 1 Jarek Nowak
http://novasamgpvm01.fnal.gov:8888/fts/status
58 2 Jarek Nowak
59 47 Pavanpoot Pandey
h1. File Transfer details (BlueArc to dCache)
60 6 Pavanpoot Pandey
61 10 Pavanpoot Pandey
Files on bluearc which is no longer in use, needs to be archived on pnfs. The first such file which is moved to pnfs is /nova/prod/mc/S13-06-05/cosmics/fd/ which is about 6.7 TB. The pnfs location of this file is /pnfs/nova/archives/2014-APR/mc/S13-06-05/cosmics/fd/xx where xx is the file number. The files on bluearc were divided into further smaller directories according to their run numbers. The sub-directory with fewer files make file operation much more efficient than putting all files together in one directory. The above file from bluearc were divided into a list of 100 sub-directories containing list of 111 files each.
62 10 Pavanpoot Pandey
63 11 Pavanpoot Pandey
h3. Copy, Verify and Remove
64 10 Pavanpoot Pandey
65 13 Pavanpoot Pandey
To do operations like copy, verify and remove a script is written which runs by putting arguments copy, verify and remove in the command line. The files to be moved from bluearc needs to be checked with pnfs whether it already exist there or not. If it already exists then its copying is skipped and copy starts for the next file. 
66 14 Pavanpoot Pandey
After copying the file to the pnfs it is verified by checking the size of the file. When the verification is done it is removed from the bluearc. The following steps are involved while doing file operations:
67 14 Pavanpoot Pandey
68 14 Pavanpoot Pandey
69 14 Pavanpoot Pandey
1. Login to any VMs (novagpvm's)
70 14 Pavanpoot Pandey
2. Go to the directory where script is written and use the following commands to get access to file operation as 'novapro'
71 28 Pavanpoot Pandey
2. ksu novapro (This gives one permission to do operations with files)
72 14 Pavanpoot Pandey
3. source novaprosource
73 14 Pavanpoot Pandey
74 15 Pavanpoot Pandey
After these steps one is ready to run scripts for file operations.
75 1 Jarek Nowak
76 19 Pavanpoot Pandey
h2. Details of files move to dCache off the bluearc
77 17 Pavanpoot Pandey
78 19 Pavanpoot Pandey
h3. S13-06-05/cosmics/fd ~ 6.7 TB
79 17 Pavanpoot Pandey
80 26 Pavanpoot Pandey
The file on Bluearc (/nova/prod/mc/S13-06-05/cosmics/fd) ~6.7 TB have been moved to dCache (/pnfs/nova/archives/2014-APR/mc/S13-06-05/cosmics/fd/xx) where xx is the file number. These files have been divided into smaller parts to make file operation easier.
81 15 Pavanpoot Pandey
82 23 Pavanpoot Pandey
h3. S14-02-05/cosmics/fd ~ 7.1 TB
83 20 Pavanpoot Pandey
84 27 Pavanpoot Pandey
Following the similar procedures the files these files are also transferred to dCache (/pnfs/nova/archives/2014-MAY/mc/B13-10-23/cosmics/fd/).
85 27 Pavanpoot Pandey
86 31 Pavanpoot Pandey
h3. B13-10-23/cosmics/fd ~ 16 TB
87 1 Jarek Nowak
88 31 Pavanpoot Pandey
Again the files from this area was split into a set of smaller files for quicker operations. Following files were removed from the BlueArc without copying it to the dCache:
89 31 Pavanpoot Pandey
/nova/prod/mc/B13-10-23/cosmics/fd/ImprovedTrans_AdamReadoutSim
90 31 Pavanpoot Pandey
/nova/prod/mc/B13-10-23/cosmics/fd/ImprovedTrans_multiplexReadoutSim
91 31 Pavanpoot Pandey
/nova/prod/mc/B13-10-23/cosmics/fd/ImprovedTrans_newMultiplexReadoutSim
92 31 Pavanpoot Pandey
93 31 Pavanpoot Pandey
Following files was copied to the dCache first and then it was removed off the BlueArc:
94 31 Pavanpoot Pandey
/pnfs/nova/archives/2014-MAY/mc/B13-10-23/cosmics/fd/.
95 31 Pavanpoot Pandey
96 31 Pavanpoot Pandey
These files were copied to: /pnfs/nova/archives/2014-APR/mc/S13-06-05/cosmics/fd/
97 31 Pavanpoot Pandey
This area has directories with run numbers for the ease of access.
98 31 Pavanpoot Pandey
99 32 Pavanpoot Pandey
h3. S13-10-11/cosmics/fd ~ 5.1 TB
100 32 Pavanpoot Pandey
101 32 Pavanpoot Pandey
The same strategy was used for this data set as well. This time the files have been kept in the sub folders of the iterations as they were in bluearc.
102 32 Pavanpoot Pandey
103 1 Jarek Nowak
/nova/prod/mc/S13-10-11/cosmics/fd/Oct10ReadoutSim_ChanMask11342
104 32 Pavanpoot Pandey
/nova/prod/mc/S13-10-11/cosmics/fd/Oct2ReadoutSim_ChanMask11342
105 33 Pavanpoot Pandey
/nova/prod/mc/S13-10-11/cosmics/fd/StaggerOct10ReadoutSim_ChanMask11342
106 10 Pavanpoot Pandey
107 34 Pavanpoot Pandey
These files are archives in dCache at:
108 34 Pavanpoot Pandey
/pnfs/nova/archives/2014-JULY/mc/S13-10-11/cosmics/fd/
109 34 Pavanpoot Pandey
110 34 Pavanpoot Pandey
It has three sub-directories:
111 34 Pavanpoot Pandey
/pnfs/nova/archives/2014-JULY/mc/S13-10-11/cosmics/fd/Oct10ReadoutSim_ChanMask11342
112 34 Pavanpoot Pandey
/pnfs/nova/archives/2014-JULY/mc/S13-10-11/cosmics/fd/Oct2ReadoutSim_ChanMask11342
113 34 Pavanpoot Pandey
/pnfs/nova/archives/2014-JULY/mc/S13-10-11/cosmics/fd/StaggerOct10ReadoutSim_ChanMask11342
114 34 Pavanpoot Pandey
115 34 Pavanpoot Pandey
116 38 Pavanpoot Pandey
The folder /pnfs/nova/archives/2014-JULY/mc/S13-10-11/cosmics/fd/Oct10ReadoutSim_ChanMask11342 is empty now because the directories 1 to 10 in /pnfs/nova/archives/2014-JULY/mc/S13-10-11/cosmics/fd/ should have been inside it. Working on to put them back to their correct location and update this space again. The google spread sheet is attached.
117 34 Pavanpoot Pandey
118 35 Pavanpoot Pandey
h3. S13-06-13/cosmics/fd ~ 5.8 TB
119 35 Pavanpoot Pandey
120 40 Pavanpoot Pandey
All the files from:
121 40 Pavanpoot Pandey
/nova/prod/mc/S13-06-13/cosmics/fd/
122 40 Pavanpoot Pandey
123 40 Pavanpoot Pandey
have been moved to:
124 40 Pavanpoot Pandey
125 40 Pavanpoot Pandey
/pnfs/nova/archives/2014-JULY/mc/S13-06-13/cosmics/fd/
126 36 Pavanpoot Pandey
127 41 Pavanpoot Pandey
The Google spread sheet is attached.
128 41 Pavanpoot Pandey
129 49 Pavanpoot Pandey
h3. /nova/ana/calibration/FarDet/ ~ (7.1) TB
130 36 Pavanpoot Pandey
131 45 Pavanpoot Pandey
All the files *pchits and *pchitstop files from /nova/ana/calibration/FarDet/S13-09-04/ are to be transferred to dCache (/pnfs/nova/archives/2014-JULY/mc/calibFiles/S13-09-04/). The files of reference directory is kept in reference directory (/pnfs/nova/archives/2014-JULY/mc/calibFiles/S13-09-04/reference) and the reference.MANGLED files are in reference.MANGLED directory(/pnfs/nova/archives/2014-JULY/mc/calibFiles/S13-09-04/reference.MANGLED). 
132 45 Pavanpoot Pandey
Since novapro can not delete these files from current bluearc (/nova/ana/calibration/FarDet/S13-09-04/) location, Gavin have to delete them to get all these free space. The Google spreadsheet is attached.
133 35 Pavanpoot Pandey
134 44 Pavanpoot Pandey
h3. S12-12-12/genie/{fd, nd} ~ 13TB
135 43 Pavanpoot Pandey
136 51 Pavanpoot Pandey
The FD and ND (/nova/prod/mc/S12-12-12/genie/) files have been archived to dCache (/pnfs/nova/archives/2014-AUG/mc/S12-12-12/genie/).
137 43 Pavanpoot Pandey
138 58 Pavanpoot Pandey
h3. mdc_S12-06-17 ~ 2.4TB
139 52 Pavanpoot Pandey
140 66 Pavanpoot Pandey
The FD and ND (/nova/ana/caf/mdc/) files have been archived to dCache (/pnfs/nova/archives/2014-Oct/mdc/). Few of the files could not be verified for its size dCache. It is copied again in /pnfs/nova/archives/2014-Oct/mdc/uncopied.
141 52 Pavanpoot Pandey
142 1 Jarek Nowak
h3. /nova/ana/caf/base/ ~ 12 TB
143 58 Pavanpoot Pandey
144 64 Pavanpoot Pandey
The FD and ND (/nova/ana/caf/base/) files is archived to dCache (/pnfs/nova/archives/2014-Oct/base/)
145 52 Pavanpoot Pandey
146 71 Pavanpoot Pandey
h3. /nova/ana/MOVED/chadj ~ 114 G (permission from Mark Messier on Oct 13, 2014)
147 68 Pavanpoot Pandey
148 72 Pavanpoot Pandey
These files have been copied to /pnfs/nova/archives/2014-Oct/MOVED/chadj/ and same directory structure have been maintained. The logs folder and the labview folders were tarred and copied to the /pnfs/nova/archives/2014-Oct/MOVED/chadj/logs and /pnfs/nova/archives/2014-Oct/MOVED/chadj/labview respectively. As these files belong to chadj novapro can't delete it. Andrew Norman has agreed to delete these. Waiting for him to delete.
149 68 Pavanpoot Pandey
150 84 Pavanpoot Pandey
h3.  /nova/prod/mc/S13-02-26 ~ 12 TB, except NDOS files
151 83 Pavanpoot Pandey
152 83 Pavanpoot Pandey
It has been archived to /pnfs/nova/archives/2014-Nov/mc/S13-02-26/.
153 83 Pavanpoot Pandey
154 96 Pavanpoot Pandey
h3. 86G	/nova/data/mc/.ToBeArchived/S12.06.17_MDC_reco
155 91 Pavanpoot Pandey
156 1 Jarek Nowak
It has been archived to /pnfs/nova/archives/2014-Nov/mc/S12.06.17_MDC_reco/.
157 96 Pavanpoot Pandey
158 99 Pavanpoot Pandey
h3. 13G	/nova/data/mc/.ToBeArchived/S12-10-04_to_FTS/
159 96 Pavanpoot Pandey
160 99 Pavanpoot Pandey
It has been archived to /pnfs/nova/archives/2014-Nov/mc/S12-10-04_to_FTS/cosmics/ndos/.
161 96 Pavanpoot Pandey
162 96 Pavanpoot Pandey
163 1 Jarek Nowak
164 1 Jarek Nowak
165 99 Pavanpoot Pandey
166 1 Jarek Nowak
h3. 2.2 TB /nova/Ana/users/cerretan to /pnfs/nova/archives/2015-May/cerretan
167 1 Jarek Nowak
168 1 Jarek Nowak
It has been archived to /pnfs/nova/archives/2015-May/cerretan.
169 100 Pavanpoot Pandey
170 100 Pavanpoot Pandey
h3. 8.7 TB	/nova/data/mc/S12-11-16/
171 100 Pavanpoot Pandey
172 100 Pavanpoot Pandey
All these files just needed a remove and have been done.
173 100 Pavanpoot Pandey
174 100 Pavanpoot Pandey
175 1 Jarek Nowak
176 1 Jarek Nowak
177 101 Pavanpoot Pandey
178 101 Pavanpoot Pandey
h3. 2.0 TB	/nova/prod/mc/S12-12-12/ to /pnfs/nova/archives/2015-June/mc/S12-12-12/
179 101 Pavanpoot Pandey
180 101 Pavanpoot Pandey
These files have been archived to /pnfs/nova/archives/2015-June/mc/S12-12-12/
181 91 Pavanpoot Pandey
182 2 Jarek Nowak
h2. Details about metadata for given production release.
183 3 Jan Zirnstein
184 5 Jan Zirnstein
h3. [[S12-10-31]]
185 5 Jan Zirnstein
186 4 Jan Zirnstein
h3. [[S12.06.17]]
187 4 Jan Zirnstein
188 3 Jan Zirnstein
h3. [[S12.02.14]]
189 3 Jan Zirnstein
190 2 Jarek Nowak
h3. [[S11.11.06]]