Support #5397
Remove /minos/data/reco_near and reco_far from Bluearc
Description
/pnfs/minos filled Sat 2014-02-08
We will remove all reco ntuples from Bluearc,
as these 53 TB of files are available in DCache.
This was discussed and approved at the January 2014 collaboration meeting
At SLF6 ( minos-slf6 ) files can be opened directly from /pnfs/minos/reco_*
On other systems they can re copied with
ifdh cp or dccp or
Overall plan :
1) remove dogwood1
2) Demonstrate and document ifdh cp access to /pnfs/minos
3) Announce a detailed schedule as noted in the Jan 2014 collaboration meeting.
3) Stage each release to DCache for immediate user access, before removal
History
#1 Updated by Arthur Kreymer about 7 years ago
- Estimated time set to 20.00 h
Removed dogwood1 files, which have not been accessed recently.
Space :
Sun Feb 9 07:28:21 CST 2014
Filesystem Size Used Avail Use% Mounted on
blue3.fnal.gov:/minos/data
231T 229T 2.6T 99% /minos/data
5.1T /minos/data/reco_near/dogwood1
1.7T /minos/data/reco_far/dogwood1
File count :
find /minos/data/reco_near/dogwood1 -type f | wc -l
5609
find /minos/data/reco_far/dogwood1 -type f | wc -l
11755
Most recent access :
NEAR
find /minos/data/reco_near/dogwood1 -type f -atime -160
/minos/data/reco_near/dogwood1/sntp_data/2007-12/N00013060_0000.spill.sntp.dogwood1.0.root
days files
160 1
194 19
195 2242
FAR
find /minos/data/reco_far/dogwood1 -type f -mtime -507
/minos/data/reco_far/dogwood1/sntp_data/2012-09/F00049990_0000.cosmic.sntp.dogwood1.0.root
/minos/data/reco_far/dogwood1/sntp_data/2012-09/F00050009_0000.cosmic.sntp.dogwood1.0.root
days files
506 0
507 2
987 11
988 185
989 281
1170 644
1600 2962
1663 11755
VERIFY FILE SIZES IN PNFS
RP=reco_near/dogwood1
cd /minos/data/${RP}
FILES=`find . type f | cut -f 2 -d /`
printf "${FILES}\n" | wc -l
5609
time {
for FILE in ${FILES} ; do
SB=`ls -l /minos/data/${RP}/${FILE} | cut -f 5 -d ' '`
SP=`ls -l /pnfs/minos/${RP}/${FILE} | cut -f 5 -d ' '`
if [ ${SB} -ne ${SP} ] ; then
echo OOPS
ls -l /minos/data/reco_near/dogwood1/${FILE}
ls -l /pnfs/minos/reco_near/dogwood1/${FILE}
fi
done
}
No problems reported
RP=reco_far/dogwood1
cd /minos/data/${RP}
FILES=`find . type f | cut -f 2 -d /`
printf "${FILES}\n" | wc -l
11755
time
real 5m36.007s
user 0m17.922s
sys 1m39.820s
date
df -h /minos/data
time rm -r /minos/data/reco_near/dogwood1
df -h /minos/data
Sun Feb 9 08:19:15 CST 2014
231T 229T 2.6T 99% /minos/data
Rates were low, about 1.5 files per second.
Proceeded to clear reco_far in the background
set nohup ; rm -r /minos/data/reco_far/dogwood1 &
#2 Updated by Arthur Kreymer about 7 years ago
- % Done changed from 0 to 10
Per discussion with Robert Hatcher, Alex Sousa and Rashid Mehdi,
will proceed to remove reco files for dogwood0 and dogwood3
and mrnt files for dogwood5
These have not been accessed recently, and should not be needed in Bluearc.
http://www-numi.fnal.gov/computing/dh/mdsum/2014/02/02.txt
1940 reco_far/dogwood0/
1731 reco_far/dogwood1/
2011 reco_far/dogwood3/
3976 reco_near/dogwood0/
5245 reco_near/dogwood1/
4075 reco_near/dogwood3/
du -sh /minos/data/reco_near/dogwood5/mrnt_data
732G /minos/data/reco_near/dogwood5/mrnt_data
du -sh /minos/data/reco_far/dogwood5/mrnt_data
128G /minos/data/reco_far/dogwood5/mrnt_data
#3 Updated by Arthur Kreymer about 7 years ago
- % Done changed from 10 to 20
The dogwood0 and dogwood3 removals are running.
See /minos/data/users/minospro/maint/dog03-2014-rm.log
A few files were not present in PNFS, these are not being removed.
FILE COUNTS
for REL in dogwood0 dogwood3 ; do
for DET in near far ; do
printf "${DET}/${REL} "
find /minos/data/reco_${DET}/${REL} -type f | wc -l
done ; done
near/dogwood0 8857
far/dogwood0 13492
near/dogwood3 4592
far/dogwood3 12310
ACCESS 200 days
for REL in dogwood0 dogwood3 ; do
for DET in near far ; do
printf "${DET}/${REL} "
find /minos/data/reco_${DET}/${REL} -type f -atime -200 | wc -l
done ; done
near/dogwood0 13
far/dogwood0 0
near/dogwood3 14
far/dogwood3 33
CHECKING PNFS
for REL in dogwood0 dogwood3 ; do
for DET in near far ; do
DR=reco_${DET}/${REL}
date
printf "${DR} "
cd /minos/data/${DR}
FILES=`find . type f | cut -f 2 -d /`
printf "${FILES}\n" | wc -l
for FILE in ${FILES} ; do
SB=`ls -l /minos/data/${DR}/${FILE} | cut -f 5 -d ' '`
SP=`ls -l /pnfs/minos/${DR}/${FILE} | cut -f 5 -d ' '`
if [ ${SB} -ne ${SP} ] ; then
echo OOPS
ls -l /minos/data/reco_near/dogwood1/${FILE}
ls -l /pnfs/minos/reco_near/dogwood1/${FILE}
exit
fi
done
done ; done 2>&1 | tee /var/tmp/d03nfscan
Mon Feb 10 16:00:05 CST 2014
reco_near/dogwood0 8857
ls: /pnfs/minos/reco_near/dogwood0/mrnt_data/2010-05/N00017947_0004.spill.mrnt.dogwood0.0.root: No such file or directory
-bash: [: 505017831: unary operator expected
ls: /pnfs/minos/reco_near/dogwood0/mrnt_data/2010-11/N00018985_0005.spill.mrnt.dogwood0.0.root: No such file or directory
-bash: [: 59217336: unary operator expected
ls: /pnfs/minos/reco_near/dogwood0/mrnt_data/2010-11/N00018988_0004.spill.mrnt.dogwood0.0.root: No such file or directory
-bash: [: 697371055: unary operator expected
ls: /pnfs/minos/reco_near/dogwood0/sntp_data/2010-05/N00017947_0004.cosmic.sntp.dogwood0.0.root: No such file or directory
-bash: [: 330803293: unary operator expected
ls: /pnfs/minos/reco_near/dogwood0/sntp_data/2010-05/N00017947_0004.spill.sntp.dogwood0.0.root: No such file or directory
-bash: [: 1279343752: unary operator expected
ls: /pnfs/minos/reco_near/dogwood0/sntp_data/2010-11/N00018985_0005.cosmic.sntp.dogwood0.0.root: No such file or directory
-bash: [: 27425892: unary operator expected
ls: /pnfs/minos/reco_near/dogwood0/sntp_data/2010-11/N00018985_0005.spill.sntp.dogwood0.0.root: No such file or directory
-bash: [: 136015021: unary operator expected
ls: /pnfs/minos/reco_near/dogwood0/sntp_data/2010-11/N00018988_0004.cosmic.sntp.dogwood0.0.root: No such file or directory
-bash: [: 333908957: unary operator expected
ls: /pnfs/minos/reco_near/dogwood0/sntp_data/2010-11/N00018988_0004.spill.sntp.dogwood0.0.root: No such file or directory
-bash: [: 1719916077: unary operator expected
Mon Feb 10 16:15:05 CST 2014
reco_far/dogwood0 13492
Mon Feb 10 16:25:18 CST 2014
reco_near/dogwood3 4592
Mon Feb 10 16:28:40 CST 2014
reco_far/dogwood3 12310
ls: /pnfs/minos/reco_far/dogwood3/sntp_data/2004-06/l.txt: No such file or directory
-bash: [: 9282: unary operator expected
ls l /minos/data/reco_far/dogwood3/sntp_data/2004-06/l.txt 1 nav e875 9282 Mar 1 2013 /minos/data/reco_far/dogwood3/sntp_data/2004-06/l.txt
-rw-r--r-
REMOVAL
Found a few files not in PNFS.
So remove only those that match :
minospro@minos27
{
for REL in dogwood0 dogwood3 ; do
for DET in near far ; do
DR=reco_${DET}/${REL}
date
printf "${DR} "
cd /minos/data/${DR}
FILES=`find . type f | cut -f 2 -d /`
printf "${FILES}\n" | wc -l
for FILE in ${FILES} ; do
SB=`ls -l /minos/data/${DR}/${FILE} | cut -f 5 -d ' '`
SP=`ls -l /pnfs/minos/${DR}/${FILE} | cut -f 5 -d ' '`
if [ ${SB} -eq ${SP} ] ; then
rm /minos/data/${DR}/${FILE}
fi
done
done ; done ; date
} 2>&1 | tee /minos/data/users/minospro/maint/dog03-2014-rm.log
Mon Feb 10 17:12:37 CST 2014
reco_near/dogwood0 8857
...
#4 Updated by Arthur Kreymer about 7 years ago
- % Done changed from 20 to 50
Full log of dogwood0/3 removal.
Files not in PNFS were noted and skipped.
Space from this removal should become available in 4 days, Fri Feb 14.
Mon Feb 10 17:12:37 CST 2014
reco_near/dogwood0 8857
ls: /pnfs/minos/reco_near/dogwood0/mrnt_data/2010-05/N00017947_0004.spill.mrnt.dogwood0.0.root: No such file or directory
-bash: [: 505017831: unary operator expected
ls: /pnfs/minos/reco_near/dogwood0/mrnt_data/2010-11/N00018985_0005.spill.mrnt.dogwood0.0.root: No such file or directory
-bash: [: 59217336: unary operator expected
ls: /pnfs/minos/reco_near/dogwood0/mrnt_data/2010-11/N00018988_0004.spill.mrnt.dogwood0.0.root: No such file or directory
-bash: [: 697371055: unary operator expected
ls: /pnfs/minos/reco_near/dogwood0/sntp_data/2010-05/N00017947_0004.cosmic.sntp.dogwood0.0.root: No such file or directory
-bash: [: 330803293: unary operator expected
ls: /pnfs/minos/reco_near/dogwood0/sntp_data/2010-05/N00017947_0004.spill.sntp.dogwood0.0.root: No such file or directory
-bash: [: 1279343752: unary operator expected
ls: /pnfs/minos/reco_near/dogwood0/sntp_data/2010-11/N00018985_0005.cosmic.sntp.dogwood0.0.root: No such file or directory
-bash: [: 27425892: unary operator expected
ls: /pnfs/minos/reco_near/dogwood0/sntp_data/2010-11/N00018985_0005.spill.sntp.dogwood0.0.root: No such file or directory
-bash: [: 136015021: unary operator expected
ls: /pnfs/minos/reco_near/dogwood0/sntp_data/2010-11/N00018988_0004.cosmic.sntp.dogwood0.0.root: No such file or directory
-bash: [: 333908957: unary operator expected
ls: /pnfs/minos/reco_near/dogwood0/sntp_data/2010-11/N00018988_0004.spill.sntp.dogwood0.0.root: No such file or directory
-bash: [: 1719916077: unary operator expected
Mon Feb 10 17:54:54 CST 2014
reco_far/dogwood0 13492
Mon Feb 10 18:29:23 CST 2014
reco_near/dogwood3 4592
Mon Feb 10 19:00:15 CST 2014
reco_far/dogwood3 12310
ls: /pnfs/minos/reco_far/dogwood3/sntp_data/2004-06/l.txt: No such file or directory
-bash: [: 9282: unary operator expected
Mon Feb 10 19:30:51 CST 2014
#5 Updated by Arthur Kreymer about 7 years ago
Space became available in /minos/data as scheduled.
More tomorrow.
MINOS27 > df -h /minos/data
Filesystem Size Used Avail Use% Mounted on
blue3.fnal.gov:/minos/data
231T 222T 9.6T 96% /minos/data
MINOS27 > date
Thu Feb 13 08:46:06 CST 2014