Support #6086
Minos candidate file removal
Description
This tracks removal of Minos candidate files from tape and disk.
History
#1 Updated by Arthur Kreymer almost 7 years ago
Date: Wed, 30 Apr 2014 16:29:33 -0500
From: Arthur Kreymer <kreymer@fnal.gov>
To: minos-users@fnal.gov
Cc: minos_authors@fnal.gov
Subject: Minos candidate file removals May 13 2014
As announced and discussed since 2012,
Minos will remove from disk and tape all the Candidate files
for all reconstruction releases, both data and simulation.
These files are no longer used, and fill about half of our tapes,
roughly 430 TB in 1.6 million files.
A migration to new media is about to start,
and we do not want to move files which we will never use.
This does not affect Minos+, as we no longer write cand files.
The file removals will start May 13, 2014.
This work is tracked in https://cdcvs.fnal.gov/redmine/issues/6086
#2 Updated by Arthur Kreymer almost 7 years ago
From http://nusoft.fnal.gov/minos/dcache/CFL/CFLSUM
KF TB rn 410 142 rf 740 66 mc 480 226 NET 1630 430
#3 Updated by Arthur Kreymer almost 7 years ago
Assessed cand files in DCache read pools mindata@minos27 mkdir /minos/app/mindata/log/candx MDC=/minos/app/mindata/log/candx MDCP=/minos/app/mindata/log/candx/mdcp Files in DCache pools curl -o ${MDCP} http://fndca.fnal.gov/dcache/files/minos wc -l ${MDCP} 343040 /minos/app/mindata/log/candx/mdcp grep cand ${MDCP} | wc -l 472 grep cand ${MDCP} > ${MDC}/mdcp-cand grep dogwood0 ${MDC}/mdcp-cand | wc -l 119 cat ${MDC}/mdcp-cand | cut -f 6,7 -d / | sort -u mcout_data/cedar_phy_bhcurv mcout_data/cedar_phy_bhhi mcout_data/cedar_phy_linfix mcout_data/dogwood5 mcout_data/elm3 reco_far/cedar reco_far/dogwood0 reco_far/R1_18_2 reco_near/cedar reco_near/dogwood0 for DIR in ${DIRS} ; do printf "%30s " ${DIR} ; grep ${DIR} ${MDC}/mdcp-cand | wc -l ; done mcout_data/cedar_phy_bhcurv 32 mcout_data/cedar_phy_bhhi 8 mcout_data/cedar_phy_linfix 16 mcout_data/dogwood5 144 mcout_data/elm3 107 reco_far/cedar 25 reco_far/dogwood0 116 reco_far/R1_18_2 1 reco_near/cedar 20 reco_near/dogwood0 3
#4 Updated by Arthur Kreymer almost 7 years ago
Surveyed cand files in the CFL complete file listing Surveyed cand files in the CFL complete file listing CFL=/nusoft/app/web/htdoc/minos/dcache/CFL/CFL CFLCAND=${MDC}/cflcand CFLCANDS=${MDC}/cflcands grep cand ${CFL} | tr -s / > ${CFLCAND} wc -l ${CFLCAND} 1646801 /minos/app/mindata/log/candx/cflcand List with just paths cat ${CFLCAND} | cut -f 9 -d ' ' > ${CFLCANDS} Files not in /cand_data/ grep -v /cand_data/ ${CFLCANDS} | grep -v BAD | wc -l 425 FPS=`grep -v /cand_data/ ${CFLCANDS} | grep -v BAD` for FP in ${FPS} ; do dirname ${FP} ; done | sort -u /pnfs/minos/alignment/neardet/d59/NewAlignment /pnfs/minos/mcout_data/cedar/far/daikon_00/L010185N /pnfs/minos/mcout_data/cedar_phy_bhhi/near/daikon_04/L010185N /pnfs/minos/mcout_data/cedar_phy_bhlo/near/daikon_04/L010185N /pnfs/minos/mcout_data/R1.14/far /pnfs/minos/mcout_data/R1.7 /pnfs/minos/reco_far/R1_17/bcnd_data/2005-05 /pnfs/minos/reco_far/R1_17/bntp_data/2005-05 /pnfs/minos/reco_far/R1_17/bnts_data/2005-05 /pnfs/minos/reco_far/R1_17/sntp_data/2005-05 /pnfs/minos/reco_far/R1_17/snts_data/2005-05 /pnfs/minos/sim_reco/far/atmos/cand /pnfs/minos/sim_reco/far/cosmic/cand grep -v '\.cand\.' ${CFLCANDS} | wc -l 177 grep '\.bcnd\.' ${CFLCANDS} | wc -l 24 Proceeding with the mainline reco_near/far mcout near/far files in cand_data and name .cand.
#5 Updated by Arthur Kreymer almost 7 years ago
Opened service desk ticket to advise Enstore admins.
RITM0100001 05/15 FYI - removal of 430 TB, 1.6 Million files from /pnfs/minos
We have started the long promised removal of Minos 'candidate' files from /pnfs/minos.
The old files are no longer in use,
and new files of this sort are no longer being produced.
I am removing files at the rate of under 5 files per second,
to avoid overloading the Enstore servers.
Please contact minos-admin if there are any operational issues.
There will be four major removals:
reco_near
reco_far
mcout_data/*/near
mcout_data/*/far
#6 Updated by Arthur Kreymer almost 7 years ago
NEARCAND=${MDC}/nearcand NEARCANDL=${MDC}/nearcand-list reco_near cand_data .cand. grep /reco_near/ ${CFLCANDS} | grep /cand_data/ | grep '\.cand\.' > ${NEARCAND} wc -l ${NEARCAND} 408710 /minos/app/mindata/log/candx/nearcand Directory summary DIRS=`cat ${NEARCAND} | cut -f 5 -d / | sort -u` for DIR in ${DIRS} ; do printf "%-25s " ${DIR} ; grep /${DIR} ${NEARCAND} | wc -l done > ${MDC}/nearcand-count Get listing of NEARCAND files cat ${NEARCAND} | while read FP ; do ls -l ${FP} ; usleep 200000 done 2>&1 | tee ${NEARCANDL} date Thu May 15 11:09:08 CDT 2014 Remove the files root@if-admin-minos MDC=/minos/app/mindata/log/candx NEARCAND=${MDC}/nearcand { date cat ${NEARCAND} | while read FP ; do echo /bin/rm -f ${FP} ; /bin/rm -f ${FP} ;usleep 200000 done date } 2>&1 | tee ${MDC}/nearcand-rm chown mindata.e875 ${MDC}/nearcand-rm Thu May 15 15:00:41 CDT 2014 /bin/rm -f /pnfs/minos/reco_near/S06-06-22-R1-22/cand_data/2005-11/N00009107_0002.cosmic.cand.S06-06-22-R1-22.0.root /bin/rm -f /pnfs/minos/reco_near/S06-06-22-R1-22/cand_data/2005-11/N00009104_0019.cosmic.cand.S06-06-22-R1-22.0.root ... /bin/rm -f /pnfs/minos/reco_near/dogwood7/cand_data/2011-11/N00021294_0009.spill.cand.dogwood7.0.root Fri May 16 20:43:09 CDT 2014
#7 Updated by Arthur Kreymer almost 7 years ago
Moving on to reco_far cand mindata@minos27 MDC=/minos/app/mindata/log/candx CFLCANDS=${MDC}/cflcands FARCAND=${MDC}/farcand FARCANDL=${MDC}/farcand-list grep /reco_far/ ${CFLCANDS} | grep /cand_data/ | grep '\.cand\.' > ${FARCAND} wc -l ${FARCAND} 728429 /minos/app/mindata/log/candx/farcand Directory summary DIRS=`cat ${FARCAND} | cut -f 5 -d / | sort -u` for DIR in ${DIRS} ; do printf "%-25s " ${DIR} ; grep /${DIR} ${FARCAND} | wc -l done > ${MDC}/farcand-count less ${MDC}/farcand-count Get listing of FARCAND files date cat ${FARCAND} | while read FP ; do ls -l ${FP} ; usleep 200000 done 2>&1 | tee ${FARCANDL} date Fri May 16 21:30:03 CDT 2014 -rw-r--r-- 1 rubin e875 9135413 Jul 3 2006 /pnfs/minos/reco_far/S06-06-22-R1-22/cand_data/2005-11/F00033083_0003.spill.cand.S06-06-22-R1-22.0.root -rw-r--r-- 1 rubin e875 87455920 Jul 3 2006 /pnfs/minos/reco_far/S06-06-22-R1-22/cand_data/2005-11/F00033083_0015.all.cand.S06-06-22-R1-22.0.root ... -rw-r--r-- 1 minospro e875 11347783153 Jan 30 2012 /pnfs/minos/reco_far/dogwood7/cand_data/2011-03/F00047421_0000.cosmic.cand.dogwood7.0.root Sun May 18 21:49:32 CDT 2014
#8 Updated by Arthur Kreymer almost 7 years ago
root@if-admin-minos MDC=/minos/app/mindata/log/candx FARCAND=${MDC}/farcand { date cat ${FARCAND} | while read FP ; do echo /bin/rm -f ${FP} ; /bin/rm -f ${FP} ;usleep 200000 done date } 2>&1 | tee ${MDC}/farcand-rm chown mindata.e875 ${MDC}/farcand-rm Mon May 19 08:26:45 CDT 2014 /bin/rm -f /pnfs/minos/reco_far/S06-06-22-R1-22/cand_data/2005-11/F00033083_0003.spill.cand.S06-06-22-R1-22.0.root /bin/rm -f /pnfs/minos/reco_far/S06-06-22-R1-22/cand_data/2005-11/F00033083_0015.all.cand.S06-06-22-R1-22.0.root ... /bin/rm -f /pnfs/minos/reco_far/dogwood7/cand_data/2011-03/F00047409_0009.cosmic.cand.dogwood7.0.root /bin/rm -f /pnfs/minos/reco_far/dogwood7/cand_data/2011-03/F00047421_0000.cosmic.cand.dogwood7.0.root Wed May 21 14:22:23 CDT 2014 Oops, ran this as mindata, not root, ran second pass as root to pick up the 10K files not yet removed grep 'cannot remove' ${MDC}/farcand-rm | cut -f 2 -d \` | cut -f 1 -d \' \ > ${MDC}/farcand-rmfail wc -l ${MDC}/farcand-rmfail 10039 /minos/app/mindata/log/candx/farcand-rmfail Wed May 21 14:37:51 CDT 2014 /bin/rm -f /pnfs/minos/reco_far/cedar_phy_bhcurv/cand_data/2004-10/F00027712_0007.all.cand.cedar_phy_bhcurv.0.root /bin/rm -f /pnfs/minos/reco_far/cedar_phy_bhcurv/cand_data/2003-07/F00018033_0000.all.cand.cedar_phy_bhcurv.0.root ... /bin/rm -f /pnfs/minos/BAD/timedog3/reco_far/dogwood3/cand_data/2008-05/F00040933_0008.cosmic.cand.dogwood3.0.root /bin/rm -f /pnfs/minos/BAD/timedog3/reco_far/dogwood3/cand_data/2008-05/F00040933_0008.spill.cand.dogwood3.0.root Wed May 21 15:27:32 CDT 2014
#9 Updated by Arthur Kreymer almost 7 years ago
MC NEAR CAND mindata@minos27 MDC=/minos/app/mindata/log/candx CFLCANDS=${MDC}/cflcands MCNEARCAND=${MDC}/mcnearcand MCNEARCANDL=${MDC}/mcnearcand-list grep /mcout_data/ ${CFLCANDS} | grep /near/ | grep /cand_data/ | grep '\.cand\.' > ${MCNEARCAND} wc -l ${MCNEARCAND} 390085 /minos/app/mindata/log/candx/mcnearcand Directory summary DIRS=`cat ${MCNEARCAND} | cut -f 5 -d / | sort -u` for DIR in ${DIRS} ; do printf "%-25s " ${DIR} ; grep /${DIR} ${MCNEARCAND} | wc -l done > ${MDC}/mcnearcand-count less ${MDC}/mcnearcand-count Get listing of MCNEARCAND files date cat ${MCNEARCAND} | while read FP ; do ls -l ${FP} ; usleep 200000 done 2>&1 | tee ${MCNEARCANDL} date Mon May 19 17:11:25 CDT 2014 -rw-r--r-- 1 rubin e875 312645083 Jul 4 2006 /pnfs/minos/mcout_data/S06-06-22-R1-22/near/cand_data/n13011003_0000_L010185.cand.S06-06-22-R1-22.root -rw-r--r-- 1 rubin e875 319099234 Jul 4 2006 /pnfs/minos/mcout_data/S06-06-22-R1-22/near/cand_data/n13011009_0000_L010185.cand.S06-06-22-R1-22.root ... rw-r--r-- 1 rubin e875 314638488 Sep 23 2006 /pnfs/minos/mcout_data/cedar/near/carrot_06/L010185/cand_data/329/n13013293_0000_L010185.cand.cedar.root -rw-r--r-- 1 rubin e875 614650647 Mar 1 2007 /pnfs/minos/mcout_data/cedar/near/daikon_00/L010185N/cand_data/112/n13011120_0009_L010185N_D00.cand.cedar.root Tue May 20 22:37:32 CDT 2014 Removed the files root@if-admin-minos MDC=/minos/app/mindata/log/candx MCNEARCAND=${MDC}/mcnearcand { date cat ${MCNEARCAND} | while read FP ; do echo /bin/rm -f ${FP} ; /bin/rm -f ${FP} ;usleep 200000 done date } 2>&1 | tee ${MDC}/mcnearcand-rm chown mindata.e875 ${MDC}/mcnearcand-rm Wed May 21 16:31:26 CDT 2014 /bin/rm -f /pnfs/minos/mcout_data/S06-06-22-R1-22/near/cand_data/n13011003_0000_L010185.cand.S06-06-22-R1-22.root /bin/rm -f /pnfs/minos/mcout_data/S06-06-22-R1-22/near/cand_data/n13011009_0000_L010185.cand.S06-06-22-R1-22.root ... /bin/rm -f /pnfs/minos/mcout_data/cedar/near/carrot_06/L010185/cand_data/329/n13013293_0000_L010185.cand.cedar.root /bin/rm -f /pnfs/minos/mcout_data/cedar/near/daikon_00/L010185N/cand_data/112/n13011120_0009_L010185N_D00.cand.cedar.root Fri May 23 00:28:21 CDT 2014 wc -l ${MDC}/mcnearcand-rm 390087 /minos/app/mindata/log/candx/mcnearcand-rm grep -v /bin/rm ${MDC}/mcnearcand-rm Wed May 21 16:31:26 CDT 2014 Fri May 23 00:28:21 CDT 2014 Count are correct, this purge is done
#10 Updated by Arthur Kreymer almost 7 years ago
MC FAR CAND mindata@minos27 MDC=/minos/app/mindata/log/candx CFLCANDS=${MDC}/cflcands MCFARCAND=${MDC}/mcfarcand MCFARCANDL=${MDC}/mcfarcand-list grep /mcout_data/ ${CFLCANDS} | grep /far/ | grep /cand_data/ | grep '\.cand\.' > ${MCFARCAND} wc -l ${MCFARCAND} 79806 /minos/app/mindata/log/candx/mcfarcand Directory summary DIRS=`cat ${MCFARCAND} | cut -f 5 -d / | sort -u` for DIR in ${DIRS} ; do printf "%-25s " ${DIR} ; grep /${DIR} ${MCFARCAND} | wc -l done > ${MDC}/mcfarcand-count less ${MDC}/mcfarcand-count Get listing of MCFARCAND files date cat ${MCFARCAND} | while read FP ; do ls -l ${FP} ; usleep 200000 done 2>&1 | tee ${MCFARCANDL} date Wed May 21 07:30:53 CDT 2014 -rw-r--r-- 1 rubin e875 342776869 Jul 12 2006 /pnfs/minos/mcout_data/S06-05-25-R1-22/far/cand_data/f21001008_0000_L010185.cand.S06-05-25-R1-22.root -rw-r--r-- 1 rubin e875 339281368 Jul 12 2006 /pnfs/minos/mcout_data/S06-05-25-R1-22/far/cand_data/f21001002_0000_L010185.cand.S06-05-25-R1-22.root ... rw-r--r-- 1 rubin e875 802536135 Jul 27 2007 /pnfs/minos/mcout_data/cedar_phy_srsafitter/far/daikon_02/CosmicMu/cand_data/112/f20011124_0001_CosmicMu_D02.cand.cedar_phy_srsafitter.root -rw-r--r-- 1 rubin e875 810706447 Jul 27 2007 /pnfs/minos/mcout_data/cedar_phy_srsafitter/far/daikon_02/CosmicMu/cand_data/112/f20011125_0008_CosmicMu_D02.cand.cedar_phy_srsafitter.root Wed May 21 13:45:15 CDT 2014 Remove the files root@if-admin-minos MDC=/minos/app/mindata/log/candx MCFARCAND=${MDC}/mcfarcand { date cat ${MCFARCAND} | while read FP ; do echo /bin/rm -f ${FP} ; /bin/rm -f ${FP} ;usleep 200000 done date } 2>&1 | tee ${MDC}/mcfarcand-rm chown mindata.e875 ${MDC}/mcfarcand-rm Fri May 23 07:10:00 CDT 2014 /bin/rm -f /pnfs/minos/mcout_data/S06-05-25-R1-22/far/cand_data/f21001008_0000_L010185.cand.S06-05-25-R1-22.root /bin/rm -f /pnfs/minos/mcout_data/S06-05-25-R1-22/far/cand_data/f21001002_0000_L010185.cand.S06-05-25-R1-22.root ... /bin/rm -f /pnfs/minos/BAD/HOTDOG5/cand_data/N00019245_0003.cosmic.cand.dogwood5.0.root /bin/rm -f /pnfs/minos/BAD/HOTDOG5/cand_data/N00019245_0003.spill.cand.dogwood5.0.root Fri May 23 17:44:44 CDT 2014 # grep -v /bin/rm ${MDC}/badcand-rm Fri May 23 14:54:20 CDT 2014 Fri May 23 17:44:44 CDT 2014 # grep /bin/rm ${MDC}/badcand-rm | wc -l 31703 The file count is correct.
#11 Updated by Arthur Kreymer almost 7 years ago
SMALLCAND FILES There are 1604 files left in small directories. Most of these are hidden behind symlinks, and/or have been mv'd without enmv, making the Complete File Listing out of skew. DATA > grep -v /pnfs/minos/reco_near/ cflcands | grep -v /pnfs/minos/reco_far/ | grep -v /pnfs/minos/mcout_data/ | grep -v /BAD/ | wc -l 1604 grep -v /pnfs/minos/reco_near/ cflcands | \ grep -v /pnfs/minos/reco_far/ | \ grep -v /pnfs/minos/mcout_data/ | \ grep -v /BAD/ > smallcand DIRS=`cat ${SMALLCAND} | cut -f 4 -d / | sort -u` for DIR in ${DIRS} ; do printf "%-25s " ${DIR} ; grep /${DIR} ${SMALLCAND} | wc -l done > ${MDC}/smallcand-count alignment 1 reco_data 1582 sim_reco 21 Get listing of SMALLCAND files date cat ${SMALLCAND} | while read FP ; do ls -l ${FP} ; usleep 200000 done 2>&1 | tee ${SMALLCANDL} date Fri May 23 18:06:00 CDT 2014 ls: /pnfs/minos/sim_reco/far/cosmic/cand/cosmic_mu_r230.cand.R0.8.0.root: No such file or directory ls: /pnfs/minos/sim_reco/far/atmos/cand/atmos_nu_r110.cand.R0.8.0.root: No such file or directory ls: /pnfs/minos/sim_reco/far/cosmic/cand/up_mu_r300.cand.R0.8.0.root: No such file or directory ls: /pnfs/minos/sim_reco/far/cosmic/cand/cosmic_mu_r213.cand.R0.8.0.root: No such file or directory ... -rw-r--r-- 1 minospro e875 113714953 Dec 14 2008 /pnfs/minos/BAD/CDATA/reco_near/cedar/c_data/2008-09/N00014871_0006.cosmic.cand.cedar.0.root -rw-r--r-- 1 minospro e875 111966429 Dec 14 2008 /pnfs/minos/BAD/CDATA/reco_near/cedar/c_data/2008-10/N00015122_0021.cosmic.cand.cedar.0.root -rw-r--r-- 1 minospro e875 449981679 Dec 14 2008 /pnfs/minos/BAD/CDATA/reco_near/cedar/c_data/2008-09/N00014871_0006.spill.cand.cedar.0.root -rw-r--r-- 1 minospro e875 111887950 Dec 14 2008 /pnfs/minos/BAD/CDATA/reco_near/cedar/c_data/2008-10/N00015122_0022.cosmic.cand.cedar.0.root ... ls: /pnfs/minos/reco_data/R1.11/cand_data/2004-09/F00027036_0003.cand.R1.11.root: No such file or directory ls: /pnfs/minos/reco_data/R1.11/cand_data/2004-09/F00027048_0003.cand.R1.11.root: No such file or directory Fri May 23 18:11:38 CDT 2014 pnfs/minos/mcout_data/R0.8.0 -> /pnfs/minos/sim_reco/far Paths here changed from cand to cand_data without enmv Will clean this up next week.
#12 Updated by Arthur Kreymer about 6 years ago
- Status changed from New to Assigned
- % Done changed from 0 to 90
Set status to Assigned.