Project

General

Profile

Support #3590

Move /minos/app data files

Added by Arthur Kreymer over 6 years ago. Updated almost 5 years ago.

Status:
New
Priority:
Normal
Start date:
03/12/2013
Due date:
03/19/2013
% Done:

0%

Estimated time:
Spent time:
Duration: 8

Description

This is to track movement of most data files from /minos/app to /minos/data.
We presently have 7 TB in app.
This should be well under 2.
Reducing the content will help with the March 21 separation
globally of /*/app from /*/data volumes in Bluearc.

History

#1 Updated by Arthur Kreymer over 6 years ago

2013 02 11

#########

Need space on /minos/app 98% in use now, of 7 TB.

http://www-numi.fnal.gov/computing/dh/masum/NOW.txt

Top 13 users ( over 100 GB )

   100 avgavril/
   432 jdejong/
   421 jjling/
   338 kopp2/     02/13
   597 loiacono/
   496 pawloski/
   135 rhatcher/
   509 rmehdi/            to 198 MB 03/14
   102 rtoner/
   276 rustem/    02/12
   213 sanjay/
   333 scavan/    03/14
   101 tinti/

This is a net 4 TB, which would be a good start if moved.

#2 Updated by Arthur Kreymer over 6 years ago

Moved rustem Feb 11

du -sm /minos/app/rustem/*

79485 /minos/app/rustem/entire
98 /minos/app/rustem/farcat
991 /minos/app/rustem/FDfiles_2008
1 /minos/app/rustem/rsync-entire.log
196206 /minos/app/rustem/sigmip

Not accessed for 794 days

find . -type f -atime -794
./entire/lock/nlock

Moved

date
cp -ax /minos/app/rustem /minos/data/users/rustem/app
date
diff -r /minos/app/rustem /minos/data/users/rustem/app
date
du -sm /minos/app/rustem /minos/data/users/rustem/app

Mon Feb 11 17:10:33 CST 2013
Mon Feb 11 18:59:38 CST 2013
Mon Feb 11 21:21:38 CST 2013
276778 /minos/app/rustem
276898 /minos/data/users/rustem/app

unalias rm
date
rm -r /minos/app/rustem
ln -s /minos/data/users/rustem/app /minos/app/rustem
date
Tue Feb 12 09:29:34 CST 2013
Tue Feb 12 09:33:30 CST 2013

Filesystem Size Used Avail Use% Mounted on
blue3.fnal.gov:/minos/app
7.5T 7.1T 435G 95% /minos/app

#3 Updated by Arthur Kreymer over 6 years ago

Moved kopp2 Feb 12


Mostly Jan 7 2010, plus a jpg from Mar 10, 2010

find . -type f | wc -l
19215

find . -type f -atime -790 # latest access 791 days ago
 date -d '791 days ago'
Tue Dec 14 17:09:57 GMT 2010

    There is no /minos/data/users/kopp,
    so shift these directly to the top there.

    minos-admin

date
cp -ax  /minos/app/kopp2 /minos/data/users/kopp
date
diff -r /minos/app/kopp2 /minos/data/users/kopp
date
du -sm  /minos/app/kopp2 /minos/data/users/kopp

Tue Feb 12 11:18:06 CST 2013
Tue Feb 12 13:57:16 CST 2013
Tue Feb 12 16:39:41 CST 2013

338909    /minos/app/kopp2
339165    /minos/data/users/kopp

unalias rm
date
rm -r   /minos/app/kopp2
ln -s                     /minos/data/users/kopp /minos/app/kopp2
date
df -h /minos/app

Wed Feb 13 14:43:22 CST 2013
Wed Feb 13 15:00:37 CST 2013
Filesystem            Size  Used Avail Use% Mounted on
blue3.fnal.gov:/minos/app
                      7.5T  6.8T  761G  91% /minos/app

#4 Updated by Arthur Kreymer over 5 years ago

Usage jumped from 97 to 98% May 26

20:40 97 %
20:45 97.5 %
20:50 98 %

Getting warnings from batch sysetem
check_quota: too little space on /minos/app (based on "df")
if-nas-0.fnal.gov:/minos/app 7.5T 7.3T 230G 98%
USED=7999872828 QUOTA=8246337209 FREE=246464381(>7500000) PERCENT=2(<3)

Found large set of mbeam.root files under

120073 /minos/app/users/loiacono/BeamDataPro

BDL=/minos/app/users/loiacono/BeamDataPro/mbeam
FILES=`find ${BDL} -type f -name \*mbeam.root`

Two of these were 'recovered', not yet in /minos/data/beam_data of /pnfs/minos/beam_data

minosraw@minos-slf6
for FILE in B100904_183016.mbeam.root B100910_001320.mbeam.root ; do
cp -a /minos/app/users/loiacono/BeamDataPro/mbeam/2010-09/${FILE} \
/pnfs/minos/beam_data/2010-09/${FILE}
done

cp: writing `/pnfs/minos/beam_data/2010-09//B100904_183016.mbeam.root': Input/output error
cp: writing `/pnfs/minos/beam_data/2010-09//B100910_001320.mbeam.root': Input/output error

root@if-admin-minos

BDL=/minos/app/users/loiacono/BeamDataPro/mbeam
FILES=`find ${BDL} -type f -name \*mbeam.root`

mkdir /minos/app/root/log/appx

find ${BDL} -type f -name \*mbeam.root -exec ls -l {} \; \
> /minos/app/root/log/appx/loiacono-mbeam

wc -l /minos/app/root/log/appx/loiacono-mbeam
1384 /minos/app/root/log/appx/loiacono-mbeam

unalias rm
for FILE in ${FILES} ; do
FD=`echo ${FILE} | cut f 8 -d /`
SD=`ls -l /minos/data/beam_data/${FD} | cut -f 5,5 -d ' '`
SL=`ls -l ${FILE} | cut -f 5,5 -d ' '`
[ "${SD}" != "${SL}" ] && ls -l ${FILE} /minos/data/beam_data/${FD}
[ "${SD}" == "${SL}" ] && rm -v ${FILE}
done 2>&1 | tee /minos/app/root/log/appx/loiacono-mbeam-rm

du -sm /minos/app/users/loiacono/BeamDataPro
20 /minos/app/users/loiacono/BeamDataPro

#5 Updated by Arthur Kreymer almost 5 years ago

Moving all 77 GB deb4

No /minos/data/users/deb4, so make and move

mkdir /minos/data/users/deb4
chown 11231 /minos/data/users/deb4

date
cp -ax /minos/app/deb4 /minos/data/users/deb4/app
date
diff -r /minos/app/deb4 /minos/data/users/deb4/app
date
Wed Nov 26 16:33:29 CST 2014
Wed Nov 26 17:01:52 CST 2014
diff: /minos/app/deb4/daikon04_mc/for_field/shw_compare/analysis/.#nu_passes.C: No such file or directory
diff: /minos/data/users/deb4/app/daikon04_mc/for_field/shw_compare/analysis/.#nu_passes.C: No such file or directory
diff: /minos/app/deb4/test.R1.26/include/MCReweight: No such file or directory
diff: /minos/data/users/deb4/app/test.R1.26/include/MCReweight: No such file or directory
diff: /minos/app/deb4/test.R1.26/include/NeugenInterface: No such file or directory
diff: /minos/data/users/deb4/app/test.R1.26/include/NeugenInterface: No such file or directory
Wed Nov 26 17:35:24 CST 2014

These are broken symlinks, OK
/minos/app/deb4/daikon04_mc/for_field/shw_compare/analysis/.#nu_passes.C ->
/minos/app/deb4/test.R1.26/include/MCReweight -> ../MCReweight
/minos/app/deb4/test.R1.26/include/NeugenInterface -> ../NeugenInterface

du -sm /minos/app/deb4 /minos/data/users/deb4/app
77710 /minos/app/deb4
77854 /minos/data/users/deb4/app

find /minos/app/deb4 -type f | wc -l
6116
find /minos/data/users/deb4/app -type f | wc -l
6116

df -m /minos/app
chmod -R 755 /minos/app/deb4
rm -r /minos/app/deb4
df -m /minos/app

Filesystem 1M-blocks Used Available Use% Mounted on
if-nas-0.fnal.gov:/minos/app 7864320 7630678 233643 98% /minos/app
if-nas-0.fnal.gov:/minos/app 7864320 7552982 311338 97% /minos/app

ln -s /minos/data/users/deb4/app /minos/app/deb4
date
Wed Nov 26 22:52:25 CST 2014



Also available in: Atom PDF