Project

General

Profile

Support #15579

/grid/fermiapp minos quota

Added by Arthur Kreymer over 2 years ago. Updated over 2 years ago.

Status:
Work in progress
Priority:
Immediate
Start date:
02/17/2017
Due date:
03/02/2017
% Done:

50%

Estimated time:
10.00 h
Duration: 14

Description

The /grid/fermiapp minos quota is full.

Items need to be removed, or moved to /minos/app.

History

#1 Updated by Arthur Kreymer over 2 years ago

  • Due date set to 02/18/2017
  • % Done changed from 0 to 10
  • Estimated time set to 3.00 h
Date: Fri, 17 Feb 2017 04:32:02 +0000 
From: Michael H Kirby <kirby@fnal.gov>
To: Andrew J. Romero <romero@fnal.gov>, Arthur E Kreymer <kreymer@fnal.gov>,
    Brian J. Rebel x8226 14112N <brebel@fnal.gov>,
    Margaret  Votava <votava@fnal.gov>
Subject: Fwd: Fermilab, file-svc-b-2,
    2017-02-16 22:18:38-06:00: Quota denied request
These keep showing up in my inbox, so I suspect that no one from Minos is getting them. Anyone know the offline production
coordinator on Minos?

Kirby

----------------------- Michael Kirby
kirby@fnal.gov
630-965-1456

Begin forwarded message:

      From: file-svc-b-2 <root@smu2.fnal.gov>
      Date: February 16, 2017 at 22:18:00 CST
      To: <fermigrid-help@fnal.gov>
      Subject: Fermilab, file-svc-b-2, 2017-02-16 22:18:38-06:00: Quota denied request

      Server Attributes:
      Server name
      file-svc-b\file-svc-b-2
      Contact
      Andy Romero 630-840-4733 romero@fnal.gov, Ray Pasetes 630-840-5250 rayp@fnal.gov
      Location
      PO Box 500 Kirk Road and Wilson Street Batavia IL 60510 USA Fermilab Backup Archival Storage Services
      Software version
      12.7.4221.09 (built 2016-08-23 15:44:46+01:00)
      Hardware version
      NAS Platform (M4SMKW1615245)
      MAC ID
      C6-17-1D-7E-A3-A0

      Quota hard limit denied request. Usage=300 GB, requested increase=64 KB (limit=300 GB).
      Volume
      fg-r5sas-b
      ViVol 
      fermigrid-fermiapp
      Target
      Group: minos-2

#2 Updated by Arthur Kreymer over 2 years ago

  • % Done changed from 10 to 20

I moved about 1.5 GB of unused parrot files to /minos/app/minsoft/parrot

This freed a bit of quota. I will search for more later today.
See ~mindata/maint/fermiapp.size from last year.

 quota -g -s
blue2:/fermigrid-fermiapp
                   300G*      0    300G          32417k       0       0        
find parrot -type f -atime -1009
SOFT > date -d '1010 days ago'
Wed May 14 09:06:52 CDT 2014

date
cp -ax /grid/fermiapp/minos/parrot /minos/app/minsoft/parrot

du -sm /grid/fermiapp/minos/parrot /minos/app/minsoft/parrot
1518    /grid/fermiapp/minos/parrot
1518    /minos/app/minsoft/parrot

diff -r  /grid/fermiapp/minos/parrot /minos/app/minsoft/parrot

  ( as mindata )

date ; rm -r /grid/fermiapp/minos/parrot
Fri Feb 17 08:16:30 CST 2017

quotaquota -g -s
blue2:/fermigrid-fermiapp
                   299G       0    300G          32412k       0       0        

#3 Updated by Arthur Kreymer over 2 years ago

  • % Done changed from 20 to 30

Report of space used currently :

quotaquota -g -s
blue2:/fermigrid-fermiapp
                   299G       0    300G          32412k       0       0        

du -sm /grid/fermiapp/minos/* 2>1 | tee /minos/app/minsoft/log/fermiapp-201702.log
sort -nr -k 1  /minos/app/minsoft/log/fermiapp-201702.log

130225    /grid/fermiapp/minos/products
91473    /grid/fermiapp/minos/minossoft
36933    /grid/fermiapp/minos/minfarm
24967    /grid/fermiapp/minos/sim
15009    /grid/fermiapp/minos/minossoft_srt_64bit
10215    /grid/fermiapp/minos/offline
1314    /grid/fermiapp/minos/kordosky
923    /grid/fermiapp/minos/VDT
50    /grid/fermiapp/minos/medeiros
25    /grid/fermiapp/minos/numisoft
5    /grid/fermiapp/minos/OSG1
...

#4 Updated by Arthur Kreymer over 2 years ago

It appears that the 10 GB in /grid/fermiapp/minos/offline
can be safely shifted to /minos/app.

Files there were most recently access in Oct 2016.

time find offline -type f -atime -200
offline/products/setup_test_root_52200b.sh
offline/minossoft/setup/setup_minossoft_FNALU_new.sh
offline/minossoft/setup/setup_minossoft_FNALU_parser.sh
offline/minossoft/setup/setup_minossoft_MINOS-RHATCHER.sh
offline/minossoft/setup/setup_minossoft_MINOS_BATCH.csh
offline/minossoft/setup/setup_minossoft_MINOS_BATCH.sh
offline/minossoft/setup/setup_minossoft_soudan
offline/minossoft/setup/setup_minossoft_soudan.sh

real    3m6.457s

ls -lu offline/products/setup_test_root_52200b.sh
-rw-r--r-- 1 rhatcher e875 596 Oct 26 15:13 offline/products/setup_test_root_52200b.sh

ls -tlur offline/minossoft/setup |  grep -v ^d
...
-rw-r--r-- 1 rhatcher e875   290 May 14  2014 setup_minossoft_FNALU_grid.sh.~2~
-rw-r--r-- 1 rhatcher e875   299 May 14  2014 setup_minossoft_FNALU_grid.csh.~2~
-rwxr-xr-x 1 rhatcher e875   407 Oct 26 15:12 setup_minossoft_MINOS_BATCH.csh
-rwxr-xr-x 1 rhatcher e875   400 Oct 26 15:12 setup_minossoft_MINOS_BATCH.sh
-rwxr-xr-x 1 rhatcher e875 10440 Oct 26 15:12 setup_minossoft_FNALU_parser.sh
-rw-r--r-- 1 rhatcher e875   299 Oct 26 15:22 setup_minossoft_FNALU_new.sh
-rw-r--r-- 1 rhatcher e875    74 Oct 26 15:23 setup_minossoft_soudan
-rw-r--r-- 1 rhatcher e875  6611 Oct 26 15:23 setup_minossoft_soudan.sh
-rw-r--r-- 1 rhatcher e875  6352 Oct 26 15:23 setup_minossoft_MINOS-RHATCHER.sh
lrwxrwxrwx 1 rhatcher e875    29 Feb 17 14:59 setup_minossoft_FNALU.csh -> setup_minossoft_FNALU_new.csh
lrwxrwxrwx 1 rhatcher e875    28 Feb 17 14:59 setup_minossoft_FNALU.sh -> setup_minossoft_FNALU_new.sh

The next most recently access files were read in 2015

ls -tlur offline/minossoft/packages/MuonRemoval/R2-0-1 |  grep -v ^d
-rw-r--r-- 1 rhatcher e875  7011 May 14  2014 StripCands.cxx
-rw-r--r-- 1 rhatcher e875 11099 Oct 14  2015 RemoveMuon.cxx
-rw-r--r-- 1 rhatcher e875  2281 Oct 14  2015 RemoveMuon.h

ls -tlur offline/minossoft/packages/MuonRemoval/R2-0-1/macros |  grep -v ^d
-rw-r--r-- 1 rhatcher e875 9252 May 14  2014 run_mre_MC_near_reroot.C
-rw-r--r-- 1 rhatcher e875 8884 Oct 14  2015 run_mrcc_DATA_near_cand_cedar.C

#5 Updated by Arthur Kreymer over 2 years ago

I have cleared a bit more space,
shifting obsolete VDT to /minos/app till we archive it properly.

find VDT -type f -atime -1010 | wc -l
0
DATA > find VDT -type f -atime -1011 | wc -l
17659

date -d '1010 days ago'
Wed May 14 17:08:43 CDT 2014

GFMV=/grid/fermiapp/minos/VDT
MAV=/minos/app/VDT

date
cp -ax  ${GFMV} ${MAV}
date
du -sm  ${GFMV} ${MAV}
diff -r ${GFMV} ${MAV}

Fri Feb 17 16:10:55 CST 2017
Fri Feb 17 16:13:40 CST 2017

924    /grid/fermiapp/minos/VDT
924    /minos/app/VDT

quota -g -s
blue2:/fermigrid-fermiapp
                   299G       0    300G          32414k       0       0        

chmod -R u+w ${GFMV}
date ; rm -r ${GFMV}
Fri Feb 17 16:17:49 CST 2017

quota -g -s
blue2:/fermigrid-fermiapp
                   298G       0    300G          32394k       0       0        

#6 Updated by Arthur Kreymer over 2 years ago

I have started a copy of /grid/fermiapp/minos/offline to /minos/app/offline.

I will edit this Issue entry as the work progresses.

    minsoft@minos-data

GFMO=/grid/fermiapp/minos/offline
MAO=/minos/app/offline

date
cp -ax  ${GFMO} ${MAO}
date
du -sm  ${GFMO} ${MAO}
diff -r ${GFMO} ${MAO}

Fri Feb 17 15:59:48 CST 2017
cp: cannot open `/grid/fermiapp/minos/offline/minossoft/packages/Production/HEAD/.swp' for reading: Permission denied
cp: cannot open `/grid/fermiapp/minos/offline/minossoft/packages/RecoBase/HEAD/.CandStripListHandle.cxx.swp' for reading: Permission denied
cp: cannot open `/grid/fermiapp/minos/offline/minossoft/packages/RecoBase/HEAD/.CandStripListHandle.h.swp' for reading: Permission denied
cp: cannot open `/grid/fermiapp/minos/offline/minossoft/releases/development/nohup.out' for reading: Permission denied
Fri Feb 17 17:38:32 CST 2017

10215    /grid/fermiapp/minos/offline
10215    /minos/app/offline

Fixed permissions, using rhatcher account.

-rw------- 1 rhatcher e875 12288 Feb 26  2006 /grid/fermiapp/minos/offline/minossoft/packages/Production/HEAD/.swp
-rw------- 1 rhatcher e875 12288 Apr 25  2003 /grid/fermiapp/minos/offline/minossoft/packages/RecoBase/HEAD/.CandStripListHandle.cxx.swp
-rw------- 1 rhatcher e875 12288 Apr 25  2003 /grid/fermiapp/minos/offline/minossoft/packages/RecoBase/HEAD/.CandStripListHandle.h.swp
-rw------- 1 rhatcher e875 115 Apr 23  2009 /grid/fermiapp/minos/offline/minossoft/releases/development/nohup.out

GFMO=/grid/fermiapp/minos/offline

FILES='
minossoft/packages/Production/HEAD/.swp
minossoft/packages/RecoBase/HEAD/.CandStripListHandle.cxx.swp
minossoft/packages/RecoBase/HEAD/.CandStripListHandle.h.swp
minossoft/releases/development/nohup.out
'

for FILE in ${FILES} ; do ls -l ${GFMO}/${FILE} ; done

    root@if-admin-minos
for FILE in ${FILES} ; do chmod 644 ${GFMO}/${FILE} ; done
chmod: changing permissions of `/grid/fermiapp/minos/offline/minossoft/packages/Production/HEAD/.swp': Read-only file system

    rhatcher@minos-data
for FILE in ${FILES} ; do chmod 644 ${GFMO}/${FILE} ; done

for FILE in ${FILES} ; do ls -l ${GFMO}/${FILE} ; done
-rw-r--r-- 1 rhatcher e875 12288 Feb 26  2006 /grid/fermiapp/minos/offline/minossoft/packages/Production/HEAD/.swp
-rw-r--r-- 1 rhatcher e875 12288 Apr 25  2003 /grid/fermiapp/minos/offline/minossoft/packages/RecoBase/HEAD/.CandStripListHandle.cxx.swp
-rw-r--r-- 1 rhatcher e875 12288 Apr 25  2003 /grid/fermiapp/minos/offline/minossoft/packages/RecoBase/HEAD/.CandStripListHandle.h.swp
-rw-r--r-- 1 rhatcher e875 115 Apr 23  2009 /grid/fermiapp/minos/offline/minossoft/releases/development/nohup.out

    back to minsoft

for FILE in ${FILES} ; do cp -ax ${GFMO}/${FILE}  ${MAO}/${FILE} ; done

    back to the diff

diff -r ${GFMO} ${MAO} 2>&1 | grep -v 'No such file or directory$'

diff: /grid/fermiapp/minos/offline/minossoft/releases/S09-07-10-R2-00/include/PackageMaintenanceSupport: recursive directory loop

lrwxrwxrwx  1 rhatcher e875     4 Sep 18  2009 PackageMaintenanceSupport -> ..///

Repaired the link, as rhatcher

exec /bin/bash
cd   /grid/fermiapp/minos/offline/minossoft/releases/S09-07-10-R2-00/include

ls -l  PackageMaintenanceSupport
lrwxrwxrwx 1 rhatcher e875 4 Sep 18  2009 PackageMaintenanceSupport -> ..//

rm     PackageMaintenanceSupport
ln -s  ../PackageMaintenanceSupport/src PackageMaintenanceSupport
ls -l  PackageMaintenanceSupport
lrwxrwxrwx 1 rhatcher e875 32 Feb 17 21:26 PackageMaintenanceSupport -> ../PackageMaintenanceSupport/src

ls -l  PackageMaintenanceSupport/
total 128
drwxr-xr-x 2 rhatcher e875  2048 Jul 10  2009 CVS
-rw-r--r-- 1 rhatcher e875  4570 May 21  2008 GlobalPackageAnalyser.py
-rw-r--r-- 1 rhatcher e875  3804 Jul 10  2009 GlobalPackageAnalyser.pyc
-rw-r--r-- 1 rhatcher e875 14459 Oct 24  2008 HtmlPageBuilder.py
-rw-r--r-- 1 rhatcher e875 12220 Jul 10  2009 HtmlPageBuilder.pyc
-rw-r--r-- 1 rhatcher e875     0 May 16  2008 __init__.py
-rw-r--r-- 1 rhatcher e875  1099 Jul 18  2008 Mailer.py
-rw-r--r-- 1 rhatcher e875  1036 Jul 10  2009 Mailer.pyc
-rw-r--r-- 1 rhatcher e875  5796 May 20  2008 PackageAnalyser.py
-rw-r--r-- 1 rhatcher e875  5321 Jul 10  2009 PackageAnalyser.pyc
-rw-r--r-- 1 rhatcher e875 13266 Oct 31  2008 PackageMaintenanceSupport.py
-rw-r--r-- 1 rhatcher e875  9012 Oct 24  2008 SupportChecker.py
-rw-r--r-- 1 rhatcher e875  6957 Jul 10  2009 SupportChecker.pyc
-rw-r--r-- 1 rhatcher e875  5378 Sep  4  2008 SupportConfirmer.py
-rw-r--r-- 1 rhatcher e875  4855 Jul 10  2009 SupportConfirmer.pyc
-rw-r--r-- 1 rhatcher e875   918 May 16  2008 TimeUtilities.py
-rw-r--r-- 1 rhatcher e875  1243 Jul 10  2009 TimeUtilities.pyc

    And as minsoft

cd /minos/app/offline/minossoft/releases/S09-07-10-R2-00/include
ls -l  PackageMaintenanceSupport
rm     PackageMaintenanceSupport
ln -s  ../PackageMaintenanceSupport/src PackageMaintenanceSupport
ls -l  PackageMaintenanceSupport
ls -l  PackageMaintenanceSupport/
total 128
drwxr-xr-x 2 minsoft e875  2048 Jul 10  2009 CVS
-rw-r--r-- 1 minsoft e875  4570 May 21  2008 GlobalPackageAnalyser.py
-rw-r--r-- 1 minsoft e875  3804 Jul 10  2009 GlobalPackageAnalyser.pyc
-rw-r--r-- 1 minsoft e875 14459 Oct 24  2008 HtmlPageBuilder.py
-rw-r--r-- 1 minsoft e875 12220 Jul 10  2009 HtmlPageBuilder.pyc
-rw-r--r-- 1 minsoft e875     0 May 16  2008 __init__.py
-rw-r--r-- 1 minsoft e875  1099 Jul 18  2008 Mailer.py
-rw-r--r-- 1 minsoft e875  1036 Jul 10  2009 Mailer.pyc
-rw-r--r-- 1 minsoft e875  5796 May 20  2008 PackageAnalyser.py
-rw-r--r-- 1 minsoft e875  5321 Jul 10  2009 PackageAnalyser.pyc
-rw-r--r-- 1 minsoft e875 13266 Oct 31  2008 PackageMaintenanceSupport.py
-rw-r--r-- 1 minsoft e875  9012 Oct 24  2008 SupportChecker.py
-rw-r--r-- 1 minsoft e875  6957 Jul 10  2009 SupportChecker.pyc
-rw-r--r-- 1 minsoft e875  5378 Sep  4  2008 SupportConfirmer.py
-rw-r--r-- 1 minsoft e875  4855 Jul 10  2009 SupportConfirmer.pyc
-rw-r--r-- 1 minsoft e875   918 May 16  2008 TimeUtilities.py
-rw-r--r-- 1 minsoft e875  1243 Jul 10  2009 TimeUtilities.pyc

Repeated the diff

date ; diff -r  ${GFMO} ${MAO} 2>&1 | grep -v 'No such file or directory$'
date

Fri Feb 17 22:32:42 CST 2017
Fri Feb 17 23:42:41 CST 2017

-----------------

    REMOVAL

   rhatcher@minos-data

exec /bin/bash # for clean prompt

    Scan for unwriteable files

find /grid/fermiapp/minos/offline ! -perm -200 -exec ls -ald {} \;
-r--r--r-- 1 rhatcher e875 30 Jun  8  2009 /grid/fermiapp/minos/offline/products/ups/prd/MINOS_ROOT/Linux2.6-GCC_3_4/v5-22-00b/.svn/prop-base/LICENSE.svn-base
...

find /grid/fermiapp/minos/offline ! -perm -200 | wc -l
20851

    verified these are all svn/ files
find /grid/fermiapp/minos/offline ! -perm -200 -exec ls -ald {} \; | grep -v svn/

find /grid/fermiapp/minos/offline ! -perm -200 -exec chmod u+w {} \;
Sat Feb 18 09:40:28 CST 2017

    Now the actual removal

quota -g -s
blue2:/fermigrid-fermiapp
                   298G       0    300G          32399k       0       0        

date ; rm -r ${GFMO} ; date
Sat Feb 18 09:43:32 CST 2017
Sat Feb 18 10:01:39 CST 2017

quota -g -s
blue2:/fermigrid-fermiapp
                   288G       0    300G          31664k       0       0        

     FIX RO FILES UNDER /minos/app/offline

    minsoft@minos-data

find /minos/app/offline ! -perm -200 | wc -l
20851

date ; find /minos/app/offline ! -perm -200 -exec chmod u+w {} \;
date
Sat Feb 18 09:52:06 CST 2017
Sat Feb 18 09:56:38 CST 2017

#7 Updated by Arthur Kreymer over 2 years ago

  • % Done changed from 30 to 50

The offline area has been copied to /minos/app/offline
and removed from /grid/fermiapp/minos.
This gets us past the immediate crisis.

We need to remove at least one more large area.
minfarm is probably a good target.
37 GB owned by rubin, minfarm and minospro.
Could change ownership to minospro and move,
with a symlink for compatibility.

#8 Updated by Arthur Kreymer over 2 years ago

  • Due date changed from 02/18/2017 to 03/02/2017
  • Estimated time changed from 3.00 h to 10.00 h

Will archive one more large area, probably the 39 GB minfarm.

We have if-admin-minos access to /grid/fermiapp/minos per RITM0526983 02/23.

Mounted as /fermiapp/minos there to not interfere with the automounter.

~root/mount-fermiappminos contains
mount blue2:/fermigrid-fermiapp/minos /fermiapp/minos \
-t nfs -o sec=sys,vers=3,rsize=32768,wsize=32768,timeo=600,proto=tcp,hard,intr,bg

date
find minfarm -printf "%u\n" | sort | uniq -c
date

Ownerships are
Mon Feb 20 09:33:55 CST 2017
      1 asousa
      2 cvson
      2 kreymer
     11 marek
      3 mindata
 791975 minfarm
    328 minospro
      1 mkiveni
   2038 mstrait
      1 psail
      1 rmehdi
    238 rubin
     58 scavan
     28 terlyga
      3 wingmc
Mon Feb 20 09:49:51 CST 2017

Checked recent modification

time find /grid/fermiapp/minos/minfarm -mtime -100
/grid/fermiapp/minos/minfarm/scripts
/grid/fermiapp/minos/minfarm/scripts/gather_runs
/grid/fermiapp/minos/minfarm/scripts/det2cat
/grid/fermiapp/minos/minfarm/scripts/AMBROSIA_submit.sh
/grid/fermiapp/minos/minfarm/scripts/gather_runs~
/grid/fermiapp/minos/minfarm/scripts/AMBROSIA_submit.sh~
/grid/fermiapp/minos/minfarm/scripts/WRAITH_nospill.sh
/grid/fermiapp/minos/minfarm/scripts/KEYGEAR_listbuilder.sh
/grid/fermiapp/minos/minfarm/scripts/runDSTMaker.sh~
/grid/fermiapp/minos/minfarm/scripts/det2cat.bckup
/grid/fermiapp/minos/minfarm/scripts/runDSTMaker.sh
/grid/fermiapp/minos/minfarm/scripts/KEYGEAR_listbuilder.sh~

real    8m6.188s



Also available in: Atom PDF