Project

General

Profile

Task #15476

Remove from persistent runs <= 247

Added by Arthur Kreymer about 3 years ago. Updated over 2 years ago.

Status:
Resolved
Priority:
High
Assignee:
Arthur Kreymer
Start date:
02/06/2017
Due date:
02/10/2017
% Done:

100%

Estimated time:
2.00 h
Spent time:
1.90 h
Duration: 5

Description

As discussed in January ,
runs in v2data and earlier are not needed in the PNFS persistent area.

This is runs <= 247.

Remove these using the archiver/transient script,
as persistent space usage iscurrently over 90%

History

#1 Updated by Arthur Kreymer about 3 years ago

Please confirm that it is OK to remove RawData runs <= 247
from /pnfs/annie/persistent/raw/raw.
The files will remain in /pnfs/annie/raw/raw.

#2 Updated by Arthur Kreymer about 3 years ago

Date: Tue, 7 Feb 2017 01:31:35 +0000
From: Benjamin Richards <>
To:
Cc: Arthur Kreymer <>, , , ,

Hi,

Sounds good to me
thanks

Ben

#3 Updated by Arthur Kreymer about 3 years ago

  • % Done changed from 10 to 90
https://cdcvs.fnal.gov/redmine/issues/15476

    GET LIST OF RUNS

ssh annieraw@annie01.fnal.gov

RUNS=$(ls /pnfs/annie/persistent/raw/v2data)

echo ${RUNS}
0058 0062 0068 0100 0101 0102 0103 0104 0105 0106 0107 0108 0109 0110 0111 0112 0113 0114 0116 0119 0120 0121 0122 0123 0124 0125 0126 0127 0128 0129 0130 0131 0132 0133 0134 0135 0136 0137 0138 0139 0140 0141 0142 0143 0144 0145 0146 0147 0148 0149 0150 0151 0152 0153 0154 0155 0156 0157 0158 0159 0160 0161 0162 0163 0164 0165 0166 0167 0168 0169 0172 0173 0174 0175 0176 0177 0178 0179 0180 0181 0182 0183 0184 0186 0188 0189 0190 0191 0192 0194 0195 0196 0197 0198 0199 0200 0203 0204 0205 0206 0207 0208 0209 0210 0211 0212 0213 0214 0215 0216 0217 0218 0219 0220 0221 0222 0223 0224 0225 0226 0227 0228 0229 0230 0231 0234 0235 0236 0237 0238 0243 0244 0245 0246 0247

echo ${RUNS} | wc -w
135

   VERIFY AGAINST RAW/RAW

RUNSR=$(ls /pnfs/annie/persistent/raw/raw | head -135)

printf "${RUNS}\n"  > /dev/shm/RUNS
printf "${RUNSR}\n" > /dev/shm/RUNSR

diff /dev/shm/RUNS /dev/shm/RUNSR

    DRY RUN

for RUN in ${RUNS} ; do ${HOME}/archiver/transient raw ${RUN} ; done

    REAL RUN

du -sh /pnfs/annie/persistent/raw/raw
31T    /pnfs/annie/persistent/raw/raw

for RUN in ${RUNS} ; do ${HOME}/archiver/transient raw ${RUN} do ; done
RUN transient raw 0058
Version 20161225 Wed Feb  8 23:09:08 CST 2017
58647    /pnfs/annie/persistent/raw/raw/0058
removed `/pnfs/annie/persistent/raw/raw/0058/RAWDataR58S139923473476680p0.root'
removed `/pnfs/annie/persistent/raw/raw/0058/RAWDataR58S139923473476680p1.root'
...
removed `/pnfs/annie/persistent/raw/raw/0247/RAWDataR247S2p97.root'
removed `/pnfs/annie/persistent/raw/raw/0247/RAWDataR247S2p98.root'
DONE Wed Feb  8 23:18:45 CST 2017

du -sh /pnfs/annie/persistent/raw/raw
11T    /pnfs/annie/persistent/raw/raw

    SUMMARY

Usage is reduced by 20 TB, about 40%

Will verify this on the web status page tomorrow.
http://fndca3a.fnal.gov/cgi-bin/du_cgi.py

#4 Updated by Arthur Kreymer about 3 years ago

Space usage remains high, in spite of file removal :

http://fndca3a.fnal.gov/cgi-bin/du_cgi.py
shows 45 TB still in use, no reduction.

Yujun - any idea why ?

#5 Updated by Arthur Kreymer over 2 years ago

  • Status changed from Assigned to Resolved
  • % Done changed from 90 to 100

Completed Feb 2017.
Space reports took a while to catch up, due to scale of removals.

Also available in: Atom PDF