Project

General

Profile

Support #17603

Remove persistent v3data for space

Added by Arthur Kreymer almost 2 years ago. Updated almost 2 years ago.

Status:
Resolved
Priority:
Immediate
Start date:
08/28/2017
Due date:
08/28/2017
% Done:

100%

Estimated time:
1.00 h
Spent time:
Duration: 1

Description

/pnfs/annie/persistent filled around Aug 25.
Remove v3data, verifying first that raw tape copy matches.

History

#1 Updated by Arthur Kreymer almost 2 years ago

  • Assignee set to Arthur Kreymer
  • Priority changed from Normal to Immediate
  • Estimated time set to 1.00 h

Date: Mon, 28 Aug 2017 15:06:46 +0000
From: Yujun Wu <>
To: Arthur Kreymer <>
Cc: "<>" <>
Subject: Annie persistent area is full

Good morning Art,

Annie experiment persitent area is 100% by now.
Earlier, it reported as: AnnieAnalysisPools: 99.97%,totalspace
52996675207168,freespace 17954737199

All Annie write to the persistent area will hang/fail
while read may be affected due to the requests may queue up.
Could you have a look on this?
You can find the space usage by individual users at:

https://fifemon-pp.fnal.gov/dashboard/db/dcache-persistent-usage-by-vo?var-VO=annie&orgId=1
http://fndca.fnal.gov/cgi-bin/space_usage_by_user_cgi.py?key=annie

#2 Updated by Arthur Kreymer almost 2 years ago

Usage grew rapidly since Aug 9.
Probably 6 TB of new files written in Gardiner.

http://fndca3a.fnal.gov/cgi-bin/space_usage_by_user_cgi.py?key=annie
User Group Name UID GID Space used (GiB) # File
annieraw annie Annie 51481 9467 40,528 89,931
gardiner annie Steven Gardiner 51313 9467 6,298 22,540
brichard annie Benjamin Richards 51115 9467 1,824 3,408
http://fndca3a.fnal.gov/cgi-bin/du_cgi.py?key=annie

/pnfs/fs/usr/annie/persistent 49,108
/pnfs/fs/usr/annie/persistent/raw/raw 18,283
/pnfs/fs/usr/annie/persistent/raw/v6data 10,239
/pnfs/fs/usr/annie/persistent/raw/v5data 7,960
/pnfs/fs/usr/annie/persistent/users/gardiner 6,300
/pnfs/fs/usr/annie/persistent/raw/v3data 3,145
/pnfs/fs/usr/annie/persistent/raw/v4data 2,337

See the previous persisistance clearance at
https://cdcvs.fnal.gov/redmine/issues/15476

#3 Updated by Arthur Kreymer almost 2 years ago

  • Due date set to 08/28/2017
  • Status changed from New to Resolved
  • % Done changed from 0 to 100

ssh

LIST OF RUNS

RUNS=$(ls /pnfs/annie/persistent/raw/v3data)

echo ${RUNS}
0248 0249 0250 0251 0252 0253 0254 0255 0256 0257 0258 0259 0260 0261 0262 0263 0264 0265 0266 0326 0327

echo ${RUNS} | wc -w
21

PREVIEW

for RUN in ${RUNS} ; do ${HOME}/archiver/transient v3data ${RUN} ; done | tee /dev/shm/v3pre

grep -v ^rm /dev/shm/v3pre
looks clean

grep OOPS /dev/shm/v3pre
... nothing ...

REAL RUN

du -sh /pnfs/annie/persistent/raw/v3data
3.1T /pnfs/annie/persistent/raw/v3data

for RUN in ${RUNS} ; do ${HOME}/archiver/transient v3data ${RUN} do ; done
RUN transient v3data 0248
Version 20161225 Mon Aug 28 14:56:42 CDT 2017
1 /pnfs/annie/persistent/raw/v3data/0248
DONE Mon Aug 28 14:56:42 CDT 2017
...
DONE Mon Aug 28 15:02:38 CDT 2017

du -sh /pnfs/annie/persistent/raw/v3data
11K /pnfs/annie/persistent/raw/v3data

SUMMARY

Usage is reduced by 3.1 TB, about 6%.

We should also remove v4data, in a separate Redmine issue.



Also available in: Atom PDF