Support #5397: Remove /minos/data/reco_near and reco_far from Bluearc - BATCH - Fermilab Redmine

Support #5397

Remove /minos/data/reco_near and reco_far from Bluearc

Added by Arthur Kreymer over 6 years ago. Updated over 6 years ago.

Status:

New

Priority:

High

Assignee:

Arthur Kreymer

Start date:

02/09/2014

Due date:

% Done:

50%

Estimated time:

20.00 h

Spent time:

5.10 h

Duration:

Description

/pnfs/minos filled Sat 2014-02-08

We will remove all reco ntuples from Bluearc,
as these 53 TB of files are available in DCache.
This was discussed and approved at the January 2014 collaboration meeting

At SLF6 ( minos-slf6 ) files can be opened directly from /pnfs/minos/reco_*
On other systems they can re copied with
ifdh cp or dccp or

Overall plan :

1) remove dogwood1
2) Demonstrate and document ifdh cp access to /pnfs/minos
3) Announce a detailed schedule as noted in the Jan 2014 collaboration meeting.
3) Stage each release to DCache for immediate user access, before removal

History

#1 Updated by Arthur Kreymer over 6 years ago

Estimated time set to 20.00 h

Removed dogwood1 files, which have not been accessed recently.

Space :

Sun Feb 9 07:28:21 CST 2014

Filesystem Size Used Avail Use% Mounted on
blue3.fnal.gov:/minos/data
231T 229T 2.6T 99% /minos/data

5.1T /minos/data/reco_near/dogwood1
1.7T /minos/data/reco_far/dogwood1

File count :

find /minos/data/reco_near/dogwood1 -type f | wc -l
5609
find /minos/data/reco_far/dogwood1 -type f | wc -l
11755

Most recent access :

NEAR

find /minos/data/reco_near/dogwood1 -type f -atime -160
/minos/data/reco_near/dogwood1/sntp_data/2007-12/N00013060_0000.spill.sntp.dogwood1.0.root

days files
160 1
194 19
195 2242

FAR

find /minos/data/reco_far/dogwood1 -type f -mtime -507
/minos/data/reco_far/dogwood1/sntp_data/2012-09/F00049990_0000.cosmic.sntp.dogwood1.0.root
/minos/data/reco_far/dogwood1/sntp_data/2012-09/F00050009_0000.cosmic.sntp.dogwood1.0.root

days files
506 0
507 2
987 11
988 185
989 281
1170 644
1600 2962
1663 11755

VERIFY FILE SIZES IN PNFS

RP=reco_near/dogwood1

cd /minos/data/${RP}
FILES=`find . ~~type f | cut -f 2~~ -d /`

printf "${FILES}\n" | wc -l
5609

time {
for FILE in ${FILES} ; do
SB=`ls -l /minos/data/${RP}/${FILE} | cut -f 5 -d ' '`
SP=`ls -l /pnfs/minos/${RP}/${FILE} | cut -f 5 -d ' '`
if [ ${SB} -ne ${SP} ] ; then
echo OOPS
ls -l /minos/data/reco_near/dogwood1/${FILE}
ls -l /pnfs/minos/reco_near/dogwood1/${FILE}
fi
done
}

No problems reported

RP=reco_far/dogwood1
cd /minos/data/${RP}
FILES=`find . ~~type f | cut -f 2~~ -d /`
printf "${FILES}\n" | wc -l
11755

time
real 5m36.007s
user 0m17.922s
sys 1m39.820s

date
df -h /minos/data
time rm -r /minos/data/reco_near/dogwood1
df -h /minos/data
Sun Feb 9 08:19:15 CST 2014
231T 229T 2.6T 99% /minos/data

Rates were low, about 1.5 files per second.
   Proceeded to clear reco_far in the background

set nohup ; rm -r /minos/data/reco_far/dogwood1 &

#2 Updated by Arthur Kreymer over 6 years ago

% Done changed from 0 to 10

Per discussion with Robert Hatcher, Alex Sousa and Rashid Mehdi,
will proceed to remove reco files for dogwood0 and dogwood3
and mrnt files for dogwood5

These have not been accessed recently, and should not be needed in Bluearc.

http://www-numi.fnal.gov/computing/dh/mdsum/2014/02/02.txt

1940 reco_far/dogwood0/
  1731 reco_far/dogwood1/
  2011 reco_far/dogwood3/

3976 reco_near/dogwood0/
  5245 reco_near/dogwood1/
  4075 reco_near/dogwood3/

du -sh /minos/data/reco_near/dogwood5/mrnt_data
732G /minos/data/reco_near/dogwood5/mrnt_data
du -sh /minos/data/reco_far/dogwood5/mrnt_data
128G /minos/data/reco_far/dogwood5/mrnt_data

#3 Updated by Arthur Kreymer over 6 years ago

% Done changed from 10 to 20

The dogwood0 and dogwood3 removals are running.

See /minos/data/users/minospro/maint/dog03-2014-rm.log

A few files were not present in PNFS, these are not being removed.

FILE COUNTS

for REL in dogwood0 dogwood3 ; do
for DET in near far ; do
printf "${DET}/${REL} "
find /minos/data/reco_${DET}/${REL} -type f | wc -l
done ; done

near/dogwood0 8857
far/dogwood0 13492
near/dogwood3 4592
far/dogwood3 12310

ACCESS 200 days

for REL in dogwood0 dogwood3 ; do
for DET in near far ; do
printf "${DET}/${REL} "
find /minos/data/reco_${DET}/${REL} -type f -atime -200 | wc -l
done ; done

near/dogwood0 13
far/dogwood0 0
near/dogwood3 14
far/dogwood3 33

CHECKING PNFS

for REL in dogwood0 dogwood3 ; do
for DET in near far ; do
DR=reco_${DET}/${REL}
date
printf "${DR} "
cd /minos/data/${DR}
FILES=`find . ~~type f | cut -f 2~~ -d /`
printf "${FILES}\n" | wc -l
for FILE in ${FILES} ; do
SB=`ls -l /minos/data/${DR}/${FILE} | cut -f 5 -d ' '`
SP=`ls -l /pnfs/minos/${DR}/${FILE} | cut -f 5 -d ' '`
if [ ${SB} -ne ${SP} ] ; then
echo OOPS
ls -l /minos/data/reco_near/dogwood1/${FILE}
ls -l /pnfs/minos/reco_near/dogwood1/${FILE}
exit
fi
done
done ; done 2>&1 | tee /var/tmp/d03nfscan

Mon Feb 10 16:00:05 CST 2014
reco_near/dogwood0 8857
ls: /pnfs/minos/reco_near/dogwood0/mrnt_data/2010-05/N00017947_0004.spill.mrnt.dogwood0.0.root: No such file or directory
-bash: [: 505017831: unary operator expected
ls: /pnfs/minos/reco_near/dogwood0/mrnt_data/2010-11/N00018985_0005.spill.mrnt.dogwood0.0.root: No such file or directory
-bash: [: 59217336: unary operator expected
ls: /pnfs/minos/reco_near/dogwood0/mrnt_data/2010-11/N00018988_0004.spill.mrnt.dogwood0.0.root: No such file or directory
-bash: [: 697371055: unary operator expected
ls: /pnfs/minos/reco_near/dogwood0/sntp_data/2010-05/N00017947_0004.cosmic.sntp.dogwood0.0.root: No such file or directory
-bash: [: 330803293: unary operator expected
ls: /pnfs/minos/reco_near/dogwood0/sntp_data/2010-05/N00017947_0004.spill.sntp.dogwood0.0.root: No such file or directory
-bash: [: 1279343752: unary operator expected
ls: /pnfs/minos/reco_near/dogwood0/sntp_data/2010-11/N00018985_0005.cosmic.sntp.dogwood0.0.root: No such file or directory
-bash: [: 27425892: unary operator expected
ls: /pnfs/minos/reco_near/dogwood0/sntp_data/2010-11/N00018985_0005.spill.sntp.dogwood0.0.root: No such file or directory
-bash: [: 136015021: unary operator expected
ls: /pnfs/minos/reco_near/dogwood0/sntp_data/2010-11/N00018988_0004.cosmic.sntp.dogwood0.0.root: No such file or directory
-bash: [: 333908957: unary operator expected
ls: /pnfs/minos/reco_near/dogwood0/sntp_data/2010-11/N00018988_0004.spill.sntp.dogwood0.0.root: No such file or directory
-bash: [: 1719916077: unary operator expected
Mon Feb 10 16:15:05 CST 2014
reco_far/dogwood0 13492
Mon Feb 10 16:25:18 CST 2014
reco_near/dogwood3 4592
Mon Feb 10 16:28:40 CST 2014
reco_far/dogwood3 12310
ls: /pnfs/minos/reco_far/dogwood3/sntp_data/2004-06/l.txt: No such file or directory
-bash: [: 9282: unary operator expected

ls l /minos/data/reco_far/dogwood3/sntp_data/2004-06/l.txt
-rw-r--r- 1 nav e875 9282 Mar 1 2013 /minos/data/reco_far/dogwood3/sntp_data/2004-06/l.txt

REMOVAL

Found a few files not in PNFS.
   So remove only those that match :

minospro@minos27

{
for REL in dogwood0 dogwood3 ; do
for DET in near far ; do
DR=reco_${DET}/${REL}
date
printf "${DR} "
cd /minos/data/${DR}
FILES=`find . ~~type f | cut -f 2~~ -d /`
printf "${FILES}\n" | wc -l
for FILE in ${FILES} ; do
SB=`ls -l /minos/data/${DR}/${FILE} | cut -f 5 -d ' '`
SP=`ls -l /pnfs/minos/${DR}/${FILE} | cut -f 5 -d ' '`
if [ ${SB} -eq ${SP} ] ; then
rm /minos/data/${DR}/${FILE}
fi
done
done ; done ; date
} 2>&1 | tee /minos/data/users/minospro/maint/dog03-2014-rm.log

Mon Feb 10 17:12:37 CST 2014
reco_near/dogwood0 8857
...

#4 Updated by Arthur Kreymer over 6 years ago

% Done changed from 20 to 50

Full log of dogwood0/3 removal.
Files not in PNFS were noted and skipped.
Space from this removal should become available in 4 days, Fri Feb 14.

Mon Feb 10 17:12:37 CST 2014
reco_near/dogwood0 8857
ls: /pnfs/minos/reco_near/dogwood0/mrnt_data/2010-05/N00017947_0004.spill.mrnt.dogwood0.0.root: No such file or directory
-bash: [: 505017831: unary operator expected
ls: /pnfs/minos/reco_near/dogwood0/mrnt_data/2010-11/N00018985_0005.spill.mrnt.dogwood0.0.root: No such file or directory
-bash: [: 59217336: unary operator expected
ls: /pnfs/minos/reco_near/dogwood0/mrnt_data/2010-11/N00018988_0004.spill.mrnt.dogwood0.0.root: No such file or directory
-bash: [: 697371055: unary operator expected
ls: /pnfs/minos/reco_near/dogwood0/sntp_data/2010-05/N00017947_0004.cosmic.sntp.dogwood0.0.root: No such file or directory
-bash: [: 330803293: unary operator expected
ls: /pnfs/minos/reco_near/dogwood0/sntp_data/2010-05/N00017947_0004.spill.sntp.dogwood0.0.root: No such file or directory
-bash: [: 1279343752: unary operator expected
ls: /pnfs/minos/reco_near/dogwood0/sntp_data/2010-11/N00018985_0005.cosmic.sntp.dogwood0.0.root: No such file or directory
-bash: [: 27425892: unary operator expected
ls: /pnfs/minos/reco_near/dogwood0/sntp_data/2010-11/N00018985_0005.spill.sntp.dogwood0.0.root: No such file or directory
-bash: [: 136015021: unary operator expected
ls: /pnfs/minos/reco_near/dogwood0/sntp_data/2010-11/N00018988_0004.cosmic.sntp.dogwood0.0.root: No such file or directory
-bash: [: 333908957: unary operator expected
ls: /pnfs/minos/reco_near/dogwood0/sntp_data/2010-11/N00018988_0004.spill.sntp.dogwood0.0.root: No such file or directory
-bash: [: 1719916077: unary operator expected
Mon Feb 10 17:54:54 CST 2014
reco_far/dogwood0 13492
Mon Feb 10 18:29:23 CST 2014
reco_near/dogwood3 4592
Mon Feb 10 19:00:15 CST 2014
reco_far/dogwood3 12310
ls: /pnfs/minos/reco_far/dogwood3/sntp_data/2004-06/l.txt: No such file or directory
-bash: [: 9282: unary operator expected
Mon Feb 10 19:30:51 CST 2014

#5 Updated by Arthur Kreymer over 6 years ago

Space became available in /minos/data as scheduled.
More tomorrow.

MINOS27 > df -h /minos/data
Filesystem Size Used Avail Use% Mounted on
blue3.fnal.gov:/minos/data
231T 222T 9.6T 96% /minos/data

MINOS27 > date
Thu Feb 13 08:46:06 CST 2014

Also available in: Atom PDF

Project

General

Profile

MINOS » BATCH

Issues

Custom queries